Re: [openstack-dev] Treating notifications as a contract (CADF)
> On 9/04/2014 Sandy Walsh wrote: > Yesterday, we had a great conversation with Matt Rutkowski from IBM, > one > of the authors of the CADF spec. > > I was having a disconnect on what CADF offers and got it clarified. > > My assumption was CADF was a set of transformation/extraction rules > for > taking data from existing data structures and defining them as > well-known things. For example, CADF needs to know "who sent this > notification". I thought CADF would give us a means to point at an > existing data structure and say "that's where you find it". > > But I was wrong. CADF is a full-on schema/data structure of its own. It > would be a fork-lift replacement for our existing notifications. This was my "aha" as well, following a similar discussion with Matt and team, but also note that they've articulated an approach for bolt-on changes that would enable CADF content in existing pipelines. (https://wiki.openstack.org/wiki/Ceilometer/blueprints/support-standard-audit-formats) > However, if your service hasn't really adopted notifications yet (green > field) or you can handle a fork-lift replacement, CADF is a good option. > There are a few gotcha's though. If you have required data that is > outside of the CADF spec, it would need to go in the "attachment" > section of the notification and that still needs a separate schema to > define it. Matt's team is very receptive to extending the spec to > include these special cases though. Agreed that Matt's team was very willing to extend, but I still wonder about having to migrate appended data from its "pre-approval" location to its "permanent" location, depending on the speed of the CADF standard update. > > Anyway, I've written up all the options (as I see them) [1] with the > advantages/disadvantages of each approach. It's just a strawman, so > bend/spindle/mutilate. Cool...will add comments there. > > Look forward to feedback! > -S > > > [1] https://wiki.openstack.org/wiki/NotificationsAndCADF > > > > > On 9/3/2014 12:30 PM, Sandy Walsh wrote: > > On 9/3/2014 11:32 AM, Chris Dent wrote: > >> On Wed, 3 Sep 2014, Sandy Walsh wrote: > >> > >>> We're chatting with IBM about CADF and getting down to specifics > on > >>> their applicability to notifications. Once I get StackTach.v3 into > >>> production I'm keen to get started on revisiting the notification > >>> format and olso.messaging support for notifications. > >>> > >>> Perhaps a hangout for those keenly interested in doing something > about this? > >> That seems like a good idea. I'd like to be a part of that. I would , too, and I would suggest that much of the Ceilometer team would > >> Unfortunately I won't be at summit but would like to contribute what > >> I can before and after. > >> > >> I took some notes on this a few weeks ago and extracted what > seemed > >> to be the two main threads or ideas the were revealed by the > >> conversation that happened in this thread: > >> > >> * At the micro level have versioned schema for notifications such > that > >>one end can declare "I am sending version X of notification > >>foo.bar.Y" and the other end can effectively deal. > > Yes, that's table-stakes I think. Putting structure around the payload > > section. > > > > Beyond type and version we should be able to attach meta information > > like public/private visibility and perhaps hints for external mapping > > (this trait -> that trait in CADF, for example). > > > >> * At the macro level standardize a packaging or envelope of all > >>notifications so that they can be consumed by very similar code. > >>That is: constrain the notifications in some way so we can also > >>constrain the consumer code. > > That's the intention of what we have now. The top level traits are > > standard, the payload is open. We really only require: message_id, > > timestamp and event_type. For auditing we need to cover Who, What, > When, > > Where, Why, OnWhat, OnWhere, FromWhere. > > To whit, I think we've made good progress in this by defining the "what is the minimum content" for PaaS service notifications and gotten agreement around https://review.openstack.org/#/c/113396/11/doc/source/format.rst for the Juno release. It's been driven by many of these same questions but is fairly narrow in scope; it defines a minimum set of content, but doesn't tackle the question of structure (beyond trait typing). The timing seems right to dig deeper. > >> These ideas serve two different purposes: One is to ensure that > >> existing notification use cases are satisfied with robustness and > >> provide a contract between two endpoints. The other is to allow a > >> fecund notification environment that allows and enables many > >> participants. > > Good goals. When Producer and Consumer know what to expect, > things are > > good ... "I know to find the Instance ID ". When the consumer > > wants to deal with a notification as a generic object,
Re: [openstack-dev] [Ceilometer] Nomination of Sandy Walsh to core team
+1. Sandy has been both helpful and insightful, plus he happens to have a good handle on the many moving parts that make up this project. :-) -Phil > -Original Message- > From: Lu, Lianhao [mailto:lianhao...@intel.com] > Sent: Sunday, December 15, 2013 5:29 PM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [Ceilometer] Nomination of Sandy Walsh to > core team > +1 in support. > > -Lianhao > > Nicolas Barcet wrote on 2013-12-13: >> +1 in support of Sandy. He is a proven contributor and reviewer and he >> brings a great business vision and experience to the team. >> >> Cheers, >> Nick >> >> >> On Wed, Dec 11, 2013 at 8:18 PM, Gordon Chung wrote: >> >> >> > To that end, I would like to nominate Sandy Walsh from Rackspace to >> > ceilometer-core. Sandy is one of the original authors of StackTach, >> and > spearheaded the original stacktach-ceilometer integration. He >> has been > instrumental in many of my codes reviews, and has >> contributed much of the > existing event storage and querying code. >> >> >> +1 in support of Sandy. the Event work he's led in Ceilometer has >> been an important feature and i think he has some valuable ideas. >> >> cheers, gordon chungopenstack, ibm software standards >> ___ OpenStack-dev >> mailing >> list OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
For multi-node deployments, implementing something like inotify would allow administrators to push configuration changes out to multiple targets using puppet/chef/etc. and have the daemons pick it up without restart. Thumbs up to that. As Tim Bell suggested, API-based enabling/disabling would allow users to update meters via script, but then there's the question of how to work out the global vs. per-project tenant settings...right now we collect specified meters for all available projects, and the API returns whatever data is stored minus filtered values. Maybe I'm missing something in the suggestion, but turning off collection for an individual project seems like it'd require some deep changes. Vijay, I'll repeat dhellmann's request: do you have more detail in another doc? :-) - Phil > -Original Message- > From: Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo) > [mailto:vijayakumar.kodam@nsn.com] > Sent: Tuesday, January 07, 2014 2:49 AM > To: OpenStack Development Mailing List (not for usage questions) > Cc: chmo...@enovance.com > Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer > From: ext Chmouel Boudjnah [mailto:chmo...@enovance.com] > Sent: Monday, January 06, 2014 2:19 PM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer > > > > > > On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata Consultancy > Ser - FI/Espoo) wrote: > > In this case, simply changing the meter properties in a configuration file > should be enough. There should be an inotify signal which shall notify > ceilometer of the changes in the config file. Then ceilometer should > automatically update the meters without restarting. > > > > Why it cannot be something configured by the admin with inotifywait(1) > command? > > > > Or this can be an API call for enabling/disabling meters which could be more > useful without having to change the config files. > > > > Chmouel. > > > > I haven't tried inotifywait() in this implementation. I need to check if it > will be > useful for the current implementation. > > Yes. API call could be more useful than changing the config files manually. > > > > Thanks, > > VijayKumar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] [QA] Slow Ceilometer resource_list CLI command
> -Original Message- > From: Tim Bell [mailto:tim.b...@cern.ch] > Sent: Monday, March 17, 2014 2:04 PM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [Ceilometer] [QA] Slow Ceilometer > resource_list CLI command > > > At CERN, we've had similar issues when enabling telemetry. Our resource-list > times out after 10 minutes when the proxies for HA assume there is no > answer coming back. Keystone instances per cell have helped the situation a > little so we can collect the data but there was a significant increase in > load on > the API endpoints. > > I feel that some reference for production scale validation would be beneficial > as part of TC approval to leave incubation in case there are issues such as > this > to be addressed. > > Tim > > > -Original Message- > > From: Jay Pipes [mailto:jaypi...@gmail.com] > > Sent: 17 March 2014 20:25 > > To: openstack-dev@lists.openstack.org > > Subject: Re: [openstack-dev] [Ceilometer] [QA] Slow Ceilometer > resource_list CLI command > > > ... > > > > Yep. At AT&T, we had to disable calls to GET /resources without any filters > on it. The call would return hundreds of thousands of > > records, all being JSON-ified at the Ceilometer API endpoint, and the result > would take minutes to return. There was no default limit > > on the query, which meant every single records in the database was > returned, and on even a semi-busy system, that meant > > horrendous performance. > > > > Besides the problem that the SQLAlchemy driver doesn't yet support > pagination [1], the main problem with the get_resources() call is > > the underlying databases schema for the Sample model is wacky, and > forces the use of a dependent subquery in the WHERE clause > > [2] which completely kills performance of the query to get resources. > > > > [1] > > > https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/ > impl_sqlalchemy.py#L436 > > [2] > > > https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/ > impl_sqlalchemy.py#L503 > > > > > The cli tests are supposed to be quick read-only sanity checks of the > > > cli functionality and really shouldn't ever be on the list of slowest > > > tests for a gate run. > > > > Oh, the test is readonly all-right. ;) It's just that it's reading hundreds > > of > thousands of records. > > > > > I think there was possibly a performance regression recently in > > > ceilometer because from I can tell this test used to normally take ~60 > > > sec. > > > (which honestly is probably too slow for a cli test too) but it is > > > currently much slower than that. > > > > > > From logstash it seems there are still some cases when the resource > > > list takes as long to execute as it used to, but the majority of runs > > > take a > long time: > > > http://goo.gl/smJPB9 > > > > > > In the short term I've pushed out a patch that will remove this test > > > from gate > > > runs: https://review.openstack.org/#/c/81036 But, I thought it would > > > be good to bring this up on the ML to try and figure out what changed > > > or why this is so slow. > > > > I agree with removing the test from the gate in the short term. Medium to > long term, the root causes of the problem (that GET > > /resources has no support for pagination on the query, there is no default > for limiting results based on a since timestamp, and that > > the underlying database schema is non-optimal) should be addressed. Gordon has introduced a blueprint https://blueprints.launchpad.net/ceilometer/+spec/big-data-sql with some fixes for individual queries but +1 to the point of looking at re-architecting the schema as an approach to fixing performance. We've also seen some gains here at HP using batch writes as well but have temporarily tabled that work in favor of getting a better-performing schema in place. - Phil > > > > Best, > > -jay > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer
> From: Ladislav Smola [mailto:lsm...@redhat.com] > Sent: Wednesday, April 16, 2014 8:37 AM > To: openstack-dev@lists.openstack.org > Subject: Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer > > No response so far, but -1 on the image element for making Ceilometer > optional. Sorry for the delayed response, Ladislov. It turns out that the mailing list was filtering out these TripleO mails for me. Let me add a little context to that -1: given that a TripleO user may not want to enable a UI layer at the undercloud level (there's a use case for using the undercloud solely for spinning up the overcloud), I think we want to support as small a footprint as possible. > > OK, so what about having variable in devtest_variables: USE_TRIPLEO_UI. > I like this approach better...in fact I will look into adding something similar into the changes I'm making to enable Ceilometer by default in the overcloud control node: https://review.openstack.org/#/c/89625/1 > It would add Undercloud Ceilometer, Tuskar-UI and Horizon. And Overcloud > SNMPd. > > Defaulted to USE_TRIPLEO_UI=1 so we have UI stuff in CI. > > How does it sound? > Perhaps specify something like "UNDERCLOUD_USE_TRIPLEO_UI" to be more specific on where this will be deployed. > > On 04/14/2014 01:31 PM, Ladislav Smola wrote: > > Hello, > > > > I am planning to add Ceilometer to Undercloud as default. Since > > Tuskar-UI uses > > it as primary source of metering samples and Tuskar should be in > > Undercloud > > as default, it made sense to me. > > > > So is my assumption correct or there are some reasons not to do this? > > > > Here are the reviews, that are adding working Undercloud Ceilometer: > > https://review.openstack.org/#/c/86915/ > > https://review.openstack.org/#/c/86917/ (depends on the template > change) > > https://review.openstack.org/#/c/87215/ > > > > Configuration for automatic obtaining of stats from all Overcloud > > nodes via. > > SNMP will follow soon. > > > > Thanks, > > Ladislav > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer
> -Original Message- > From: Ladislav Smola [mailto:lsm...@redhat.com] > Sent: Wednesday, April 23, 2014 4:29 AM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer > > Hi Neal, thanks for response. > > So I would see it as UNDERCLOUD_USE_UI (TripleO UI can be placed only to > Undercloud) > > And for overcloud: OVERCLOUD_USE_UI and > OVERCLOUD_USE_CEILOMETER, cause in > overcloud users might not want UI, but only data for billing. Does it > sound reasonable? Yep, agreed that UI/metering are different use cases. Will work OVERCLOUD_USE_CEILOMETER use case into changes first, then address UI later if someone else hasn't picked it up. - Phil > > On 04/22/2014 06:23 PM, Neal, Phil wrote: > >> From: Ladislav Smola [mailto:lsm...@redhat.com] > >> Sent: Wednesday, April 16, 2014 8:37 AM > >> To: openstack-dev@lists.openstack.org > >> Subject: Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer > >> > >> No response so far, but -1 on the image element for making Ceilometer > >> optional. > > Sorry for the delayed response, Ladislov. It turns out that the mailing list > was filtering out these TripleO mails for me. > > > > Let me add a little context to that -1: given that a TripleO user may not > > want > to enable a UI layer at the undercloud level (there's a use case for using the > undercloud solely for spinning up the overcloud), I think we want to support > as small a footprint as possible. > > > >> OK, so what about having variable in devtest_variables: USE_TRIPLEO_UI. > >> > > I like this approach better...in fact I will look into adding something > > similar > into the changes I'm making to enable Ceilometer by default in the overcloud > control node: https://review.openstack.org/#/c/89625/1 > > > >> It would add Undercloud Ceilometer, Tuskar-UI and Horizon. And > Overcloud > >> SNMPd. > >> > >> Defaulted to USE_TRIPLEO_UI=1 so we have UI stuff in CI. > >> > >> How does it sound? > >> > > Perhaps specify something like "UNDERCLOUD_USE_TRIPLEO_UI" to be > more specific on where this will be deployed. > >> On 04/14/2014 01:31 PM, Ladislav Smola wrote: > >>> Hello, > >>> > >>> I am planning to add Ceilometer to Undercloud as default. Since > >>> Tuskar-UI uses > >>> it as primary source of metering samples and Tuskar should be in > >>> Undercloud > >>> as default, it made sense to me. > >>> > >>> So is my assumption correct or there are some reasons not to do this? > >>> > >>> Here are the reviews, that are adding working Undercloud Ceilometer: > >>> https://review.openstack.org/#/c/86915/ > >>> https://review.openstack.org/#/c/86917/ (depends on the template > >> change) > >>> https://review.openstack.org/#/c/87215/ > >>> > >>> Configuration for automatic obtaining of stats from all Overcloud > >>> nodes via. > >>> SNMP will follow soon. > >>> > >>> Thanks, > >>> Ladislav > >>> > >>> > >>> ___ > >>> OpenStack-dev mailing list > >>> OpenStack-dev@lists.openstack.org > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >> ___ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Adding pylint checking of new ceilometer patches
> From: Dina Belova [mailto:dbel...@mirantis.com] > On Friday, October 03, 2014 2:53 AM > > Igor, > > Personally this idea looks really nice to me, as this will help to avoid > strange code being merged and not found via reviewing process. > > Cheers, > Dina > > On Fri, Oct 3, 2014 at 12:40 PM, Igor Degtiarov > wrote: > Hi folks! > > I try too guess do we need in ceilometer checking new patches for > critical errors with pylint? > > As far as I know Nova and Sahara and others have such check. Actually > it is not checking of all project but comparing of the number of > errors without new patch and with it, and if diff is more then 0 then > patch are not taken. Looking a bit deeper it seems that Nova struggled with false positives and resorted to https://review.openstack.org/#/c/28754/ , which layers some historical checking of git on top of pylint's tendency to check only the latest commit. I can't say I'm too deeply versed in the code, but it's enough to make me wonder if we want to go that direction and avoid the issues altogether? > > I have taken as pattern Sahara's solution and proposed a patch for > ceilometer: > https://review.openstack.org/#/c/125906/ > > Cheers, > Igor Degtiarov > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > Best regards, > Dina Belova > Software Engineer > Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] unable to collect compute.node.cpu.* data
Frank, I'd echo Hang Liu's suggestion, but also encourage you to take this question to the general OpenStack mailing list (see https://wiki.openstack.org/wiki/Mailing_Lists). - Phil From: Hang H Liu [mailto:hang...@cn.ibm.com] Sent: Wednesday, November 05, 2014 7:36 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [ceilometer] unable to collect compute.node.cpu.* data Same steps I can get below results. You may need debug into get_meters() in ceilometer/storage/impl_sqlalchemy.py to see if some filters are taking effect. localadmin@ostest2:~/devstack$ ceilometer meter-list +-++-+--+--+--+ | Name | Type | Unit | Resource ID | User ID | Project ID | +-++-+--+--+--+ | compute.node.cpu.frequency | gauge | MHz | ostest2_ostest2 | None | None | | compute.node.cpu.idle.percent | gauge | % | ostest2_ostest2 | None | None | | compute.node.cpu.idle.time | cumulative | ns | ostest2_ostest2 | None | None | | compute.node.cpu.iowait.percent | gauge | % | ostest2_ostest2 | None | None | | compute.node.cpu.iowait.time | cumulative | ns | ostest2_ostest2 | None | None | | compute.node.cpu.kernel.percent | gauge | % | ostest2_ostest2 | None | None | | compute.node.cpu.kernel.time | cumulative | ns | ostest2_ostest2 | None | None | | compute.node.cpu.percent | gauge | % | ostest2_ostest2 | None | None | | compute.node.cpu.user.percent | gauge | % | ostest2_ostest2 | None | None | | compute.node.cpu.user.time | cumulative | ns | ostest2_ostest2 | None "Lu, Lianhao" 写于 2014/11/05 15:23:14: > From: "Lu, Lianhao" > To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 2014/11/05 15:31 > Subject: Re: [openstack-dev] [ceilometer] unable to collect > compute.node.cpu.* data > > Hi Frank, > > Could you try ‘celometer sample-list’ to see if the compute.node.cpu > samples are there? > > -Lianhao > > From: Du Jun [mailto:dj199...@gmail.com] > Sent: Wednesday, November 05, 2014 3:44 AM > To: openstack-dev@lists.openstack.org > Subject: [openstack-dev] [ceilometer] unable to collect > compute.node.cpu.* data > > Hi all, > > I attempt to collect compute.node.cpu as the following link mentions: > > http://docs.openstack.org/developer/ceilometer/measurements.html#compute-nova > > I set: > > compute_monitors = ComputeDriverCPUMonitor > > in /etc/nova/nova.conf and restart nova-compute, nova-scheduler, > ceilometer-agent-notification, ceilometer-api, ceilometer-collector. > > From ceilometer-agent-notification's log, I can see agent transform > and publish data samples compute.node.cpu.* > > What's more, from ceilometer database, I can see all the meters > compute.node.cpu.* > > mysql> select * from meter; > ++-++---+ > | id | name | type | unit | > ++-++---+ > | 39 | compute.node.cpu.frequency | gauge | MHz | > | 41 | compute.node.cpu.idle.percent | gauge | % | > | 38 | compute.node.cpu.idle.time | cumulative | ns | > | 45 | compute.node.cpu.iowait.percent | gauge | % | > | 42 | compute.node.cpu.iowait.time | cumulative | ns | > | 36 | compute.node.cpu.kernel.percent | gauge | % | > | 44 | compute.node.cpu.kernel.time | cumulative | ns | > | 37 | compute.node.cpu.percent | gauge | % | > | 43 | compute.node.cpu.user.percent | gauge | % | > | 40 | compute.node.cpu.user.time | cumulative | ns | > > > However, when I type > > ceilometer meter-list > > It shows nothing about
Re: [openstack-dev] [Ceilometer] Notifications from non-local exchanges
Greetings all, I'm looking at how to expand the ability of our CM instance to consume notifications and have a quick question about the configuration and flow... For the notifications central agent , we rely on the services (i.e. glance, cinder) to drop messages on the same messaging host as used by Ceilometer. From there the listener picks it up and cycles through the plugin logic to convert it to a sample. It's apparent that we can't pass an alternate hostname via the control_exchange values, so is there another method for harvesting messages off of other instances (e.g. another compute node)? - Phil ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Notifications from non-local exchanges
> -Original Message-
> From: Sandy Walsh [mailto:sandy.wa...@rackspace.com]
> Sent: Thursday, October 10, 2013 6:20 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Ceilometer] Notifications from non-local
> exchanges
>
>
> On 10/10/2013 06:16 PM, Neal, Phil wrote:
>> Greetings all, I'm looking at how to expand the ability of our CM
>> instance to consume notifications and have a quick question about
>> the configuration and flow...
>>
>> For the notifications central agent , we rely on the services (i.e.
>> glance, cinder) to drop messages on the same messaging host as used
>> by Ceilometer. From there the listener picks it up and cycles through
>> the plugin logic to convert it to a sample. It's apparent that we
>> can't pass an alternate hostname via the control_exchange values, so
>> is there another method for harvesting messages off of other
>> instances (e.g. another compute node)?
>
> Hey Phil,
>
> You don't really need to specify the exchange name to consume
> notifications. It will default to the control-exchange if not specified
> anyway.
>
> How it works isn't so obvious.
>
> Depending on the priority of then notification the oslo notifier will
> publish on . using the service's control-exchange. If
> that queue doesn't exist it'll create it and bind the control-exchange
> to it. This is so we can publish even if there are no consumers yet.
I think the common default is notifications.info, yes?
>
> Oslo.rpc creates a 1:1 mapping of routing_key and queue to topic (no
> wildcards). So we get
>
> > -> binding: routing_key "." ->
> queue "."
>
> (essentially, 1 queue per priority)
>
> Which is why, if you want to enable services to generate notifications,
> you just have to set the driver and the topic(s) to publish on. Exchange
> is implied and routing key/queue are inferred from topic.
Yep, following up to this point: Oslo takes care of the setup of exchanges on
behalf of the
services. When, say, Glance wants to push notifications onto the message bus,
they can set
the control_exchange value and the driver ("rabbit", for example) and voila! An
exchange is
set up with a default queue bound to the key.
>
> Likewise we only have to specify the queue name to consume, since we
> only need an exchange to publish.
Here's where my gap is: the notification plugins seem to assume that Ceilometer
is sitting on the same messaging node/endpoint as the service. The config file
allows
us to specify the exchange names for the services , but not endpoints, so if
Glance
is publishing to "notifications.info" on "rabbit.glance.hpcloud.net", and
ceilometer
is publishing/consuming from the "rabbit.ceil.hpcloud.net" node then the
Glance
notifications won't be collected.
I took another look at the Ceilometer config options..."rabbit_hosts"
takes multiple hosts (i.e. rabbit.glance.hpcloud.net:,
rabbit.ceil.hpcloud.net:)
but it's not clear whether that's for publishing, collection, or both? The
impl_kombu
module does cycle through that list to create the connection pool, but it's not
clear to me how it all comes together in the plugin instantiation...
>
> I have a bare-bones oslo notifier consumer and client here if you want
> to mess around with it (and a bare-bones kombu version in the parent).
Will take a look!
>
> https://github.com/SandyWalsh/amqp_sandbox/tree/master/oslo
>
> Not sure if that answered your question or made it worse? :)
>
> Cheers
> -S
>
>
>>
>>
>> - Phil
>>
>> ___ OpenStack- dev mailing
>> list OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [qa] Ceilometer ERRORS in normal runs
Sean, we currently have a BP out there to investigate basic tempest integration and I think this might fall under the same umbrella. Unfortunately I've not been able to free up my development time for it, but I've assigned it out to someone who can take a look and report back. https://blueprints.launchpad.net/tempest/+spec/basic-tempest-integration-for-ceilometer - Phil > -Original Message- > From: Sean Dague [mailto:s...@dague.net] > Sent: Sunday, October 20, 2013 7:39 AM > To: OpenStack Development Mailing List > Subject: [openstack-dev] [ceilometer] [qa] Ceilometer ERRORS in normal > runs > > Dave Kranz has been building a system so that we can ensure that during > a Tempest run services don't spew ERRORs in the logs. Eventually, we're > going to gate on this, because there is nothing that Tempest does to the > system that should cause any OpenStack service to ERROR or stack trace > (Errors should actually be exceptional events that something is wrong > with the system, not regular events). > > Ceilometer is currently one of the largest offenders in dumping ERRORs > in the gate - > http://logs.openstack.org/68/52768/1/check/check-tempest-devstack-vm- > full/76f83a4/console.html#_2013-10-19_14_51_51_271 > (that item isn't in our whitelist yet, so you'll see a lot of it at the > end of every run) > > and > http://logs.openstack.org/68/52768/1/check/check-tempest-devstack-vm- > full/76f83a4/logs/screen-ceilometer-collector.txt.gz?level=TRACE > for full details > > This seems like something is wrong in the integration, and would be > really helpful if we could get ceilometer eyes on this one to put ceilo > into a non erroring state. > > -Sean > > -- > Sean Dague > http://dague.net > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer] Changing pollster/pipeline parameters
A couple of related questions on managing CM pollster behavior via "config" type files: 1. I'm trying to make some modifications to the timing of the Glance (image) polling in a default CM install. It looks like the pipeline interval fields are the way to do it, and that I should tweak it using a yaml config file, but I can't seem to verify based on a read-through of the code. Can anyone confirm? 2. Similarly, it looks like disabling pollsters is done via the oslo.cfg logic in the agent manager. I'd like to populate that using a config fileis there logic already to do that that I haven't come across yet? - Phil ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Changing pollster/pipeline parameters
> -Original Message-
> From: Eoghan Glynn [mailto:egl...@redhat.com]
> Sent: Thursday, July 04, 2013 3:36 AM
> To: OpenStack Development Mailing List; Neal, Phil
> Subject: Re: [openstack-dev] [Ceilometer] Changing pollster/pipeline
> parameters
>
>
> Hi Phil,
>
> Some more thoughts inline ...
>
> > > A couple of related questions on managing CM pollster behavior via
> "config"
> > > type files:
> > >
> > > 1. I'm trying to make some modifications to the timing of the Glance
> > > (image)
> > > polling in a default CM install. It looks like the pipeline interval
> > > fields
> > > are the way to do it, and that I should tweak it using a yaml config file,
> > > but I can't seem to verify based on a read-through of the code. Can
> anyone
> > > confirm?
> >
> > Yep, the configured interval in the /etc/ceilometer/pipeline.yaml is the way
> > to go.
> >
> > For example:
> >
> > $ sed -i 's/interval: .*$/interval: 10/' /etc/ceilometer/pipeline.yaml
> > $ # restart ceilometer-agent-central
> > $ ceilometer sample-list -m image.size | tail -5
> > | df52ee46-91c0-417a-a27c-ca0b447fb5db | image.size | gauge |
> 25165824.0 | B
> > | | 2013-07-03T21:40:15|
> > | df52ee46-91c0-417a-a27c-ca0b447fb5db | image.size | gauge |
> 25165824.0 | B
> > | | 2013-07-03T21:40:25|
> > | df52ee46-91c0-417a-a27c-ca0b447fb5db | image.size | gauge |
> 25165824.0 | B
> > | | 2013-07-03T21:40:35|
> > | df52ee46-91c0-417a-a27c-ca0b447fb5db | image.size | gauge |
> 25165824.0 | B
> > | | 2013-07-03T21:40:45|
> >
> > +--++---++--+--
> --+
> >
> > As you see the meter is now being collected at a 10s cadence.
> >
> > This timing is pulled in via the AgentManager base class:
> >
> >
> https://github.com/openstack/ceilometer/blob/master/ceilometer/agent.p
> y#L90
>
> I should have been more explicit here, in the sense of pointing out that:
>
> (a) this pipeline config can be made specific to the central agent
> (i.e. the agent responsible for polling glance among other things)
> by adding the line:
>
> pipeline_cfg_file = pipeline-central.yaml
>
> to the ceilometer.conf picked up by the central agent (so that the
> central agent pipeline is independent of the pipeline used by the
> other agents)
>
>
> (b) multiple pipelines can be defined in a single yaml file, with different
> polling intervals for different groups of meters.
>
> For example, to make only the glance polling interval 120s while keeping
> the other central agent polling cycles at 60s, use something like the
> following in the appropriate pipeline.yaml file:
>
> ---
> -
> name: meter_pipeline
> interval: 60
> counters:
> - "*"
> - "!image"
> - "!image.size"
> transformers:
> publishers:
> - rpc://
> -
> name: image_pipeline
> interval: 120
> counters:
> - "image"
> - "image.size"
> transformers:
> publishers:
> - rpc://
>
Makes sense on paper. :=) Will try implementing today.
>
> > > 2. Similarly, it looks like disabling pollsters is done via the oslo.cfg
> > > logic in the agent manager. I'd like to populate that using a config
> > > fileis there logic already to do that that I haven't come across yet?
> >
> > Well the way I'd disable a pollster is simply by configuring the counters
> > exclusion in the pipeline.yaml.
> >
> > For example to disable the pollster providing the image.size meter referred
> > to above, the following will do the trick:
> >
> > counters:
> > - "*"
> > - "!image.size"
> >
> > i.e. allow everything but the image.size meter.
> >
> > Confirm as before using:
> >
> > $ # restart ceilometer-agent-central
> > $ ceilometer sample-list -m image.size | tail -5
>
>
> It also occurred to me that you may have been thinking of the now-obsolete
> disabled_{central|compute|...}_pollsters config options, which are no longer
> supported as they've been superceeded by the newer pipeline config.
>
Yes, that's the line of logic I was headed down. Thanks for noting the
obsolescence.
> Hope this helps clarify things for you.
>
> Cheers,
> Eoghan
>
> >
Truly helpfulthanks again!
> > > - Phil
> > >
> > > ___
> > > OpenStack-dev mailing list
> > > OpenStack-dev@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] [ceilometer] Periodic Auditing In Glance
I'm a little concerned that a batch payload won't align with "exists" events
generated from other services. To my recollection, Cinder, Trove and Neutron
all emit exists events on a per-instance basisa consumer would have to
figure out a way to handle/unpack these separately if they needed a granular
feed. Not the end of the world, I suppose, but a bit inconsistent.
And a minor quibble: batching would also make it a much bigger issue if a
consumer missed a notificationthough I guess you could counteract that by
increasing the frequency (but wouldn't that defeat the purpose?)
>
>
>
> On 08/13/2013 04:35 PM, Andrew Melton wrote:
> >> I'm just concerned with the type of notification you'd send. It has to
> >> be enough fine grained so we don't lose too much information.
> >
> > It's a tough situation, sending out an image.exists for each image with
> > the same payload as say image.upload would likely create TONS of traffic.
> > Personally, I'm thinking about a batch payload, with a bare minimum of the
> > following values:
> >
> > 'payload': [{'id': 'uuid1', 'owner': 'tenant1', 'created_at':
> > 'some_date', 'size': 1},
> >{'id': 'uuid2', 'owner': 'tenant2', 'created_at':
> > 'some_date', 'deleted_at': 'some_other_date', 'size': 2}]
> >
> > That way the audit job/task could be configured to emit in batches which
> > a deployer could tweak the settings so as to not emit too many messages.
> > I definitely welcome other ideas as well.
>
> Would it be better to group by tenant vs. image?
>
> One .exists per tenant that contains all the images owned by that tenant?
>
> -S
>
>
> > Thanks,
> > Andrew Melton
> >
> >
> > On Tue, Aug 13, 2013 at 4:27 AM, Julien Danjou > > wrote:
> >
> > On Mon, Aug 12 2013, Andrew Melton wrote:
> >
> > > So, my question to the Ceilometer community is this, does this
> > sound like
> > > something Ceilometer would find value in and use? If so, would this be
> > > something
> > > we would want most deployers turning on?
> >
> > Yes. I think we would definitely be happy to have the ability to drop
> > our pollster at some time.
> > I'm just concerned with the type of notification you'd send. It has to
> > be enough fine grained so we don't lose too much information.
> >
> > --
> > Julien Danjou
> > // Free Software hacker / freelance consultant
> > // http://julien.danjou.info
> >
> >
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer] Correct way to disable specific event collection by the collector
Greetings team, I'm working on getting a very streamlined set of collections running and I'd like to disable all notifications except Glance. It's clear that the desired event types are defined in the plugins, but I can't seem to work out how to force the collector service to load only specific handlers in the "ceilometer.collector" namespace. I *thought* it could be accomplished by editing /ceilometer/setup.cfg, but removing the entry points there didn't seem to work (the extensions manager still picks them up). Can someone give me a rough idea of how to do this? - Phil ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
