Re: [openstack-dev] [ceilometer] unable to collect compute.node.cpu.* data

2014-11-06 Thread Neal, Phil 
Frank, I'd echo Hang Liu's suggestion, but also encourage you to take this 
question to the general OpenStack mailing 
list (see https://wiki.openstack.org/wiki/Mailing_Lists). 

- Phil

From: Hang H Liu [mailto:hang...@cn.ibm.com] 
Sent: Wednesday, November 05, 2014 7:36 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [ceilometer] unable to collect compute.node.cpu.* 
data

Same steps I can get below results.  
You may need debug into get_meters() in ceilometer/storage/impl_sqlalchemy.py 
to see if some filters are taking effect.


localadmin@ostest2:~/devstack$ ceilometer meter-list
+-++-+--+--+--+
| Name                            | Type       | Unit    | Resource ID          
                | User ID                          | Project ID                 
      |
+-++-+--+--+--+
| compute.node.cpu.frequency      | gauge      | MHz     | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.idle.percent   | gauge      | %       | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.idle.time      | cumulative | ns      | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.iowait.percent | gauge      | %       | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.iowait.time    | cumulative | ns      | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.kernel.percent | gauge      | %       | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.kernel.time    | cumulative | ns      | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.percent        | gauge      | %       | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.user.percent   | gauge      | %       | ostest2_ostest2      
                | None                             | None                       
      |
| compute.node.cpu.user.time      | cumulative | ns      | ostest2_ostest2      
                | None                      



Lu, Lianhao lianhao...@intel.com 写于 2014/11/05 15:23:14:

 From: Lu, Lianhao lianhao...@intel.com
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: 2014/11/05 15:31
 Subject: Re: [openstack-dev] [ceilometer] unable to collect 
 compute.node.cpu.* data
 
 Hi Frank,
  
 Could you try ‘celometer sample-list’ to see if the compute.node.cpu
 samples are there?
  
 -Lianhao
  
 From: Du Jun [mailto:dj199...@gmail.com] 
 Sent: Wednesday, November 05, 2014 3:44 AM
 To: openstack-dev@lists.openstack.org
 Subject: [openstack-dev] [ceilometer] unable to collect 
 compute.node.cpu.* data
  
 Hi all,
  
 I attempt to collect compute.node.cpu as the following link mentions:
  
 http://docs.openstack.org/developer/ceilometer/measurements.html#compute-nova
  
 I set:
  
 compute_monitors = ComputeDriverCPUMonitor
  
 in /etc/nova/nova.conf and restart nova-compute, nova-scheduler, 
 ceilometer-agent-notification, ceilometer-api, ceilometer-collector. 
  
 From ceilometer-agent-notification's log, I can see agent transform 
 and publish data samples compute.node.cpu.*
  
 What's more, from ceilometer database, I can see all the meters 
 compute.node.cpu.*
  
 mysql select * from meter;
 ++-++---+
 | id | name                            | type       | unit      |
 ++-++---+
 | 39 | compute.node.cpu.frequency      | gauge      | MHz       |
 | 41 | compute.node.cpu.idle.percent   | gauge      | %         |
 | 38 | compute.node.cpu.idle.time      | cumulative | ns        |
 | 45 | compute.node.cpu.iowait.percent | gauge      | %         |
 | 42 | compute.node.cpu.iowait.time    | cumulative | ns        |
 | 36 | compute.node.cpu.kernel.percent | gauge      | %         |
 | 44 | compute.node.cpu.kernel.time    | cumulative | ns        |
 | 37 | compute.node.cpu.percent        | gauge      | %         |
 | 43 | compute.node.cpu.user.percent   | gauge      | %         |
 | 40 | compute.node.cpu.user.time      | cumulative | ns        |
  
  
 However, when I type
  
 ceilometer meter-list
  
 It shows nothing

Re: [openstack-dev] [Ceilometer] Adding pylint checking of new ceilometer patches

2014-10-03 Thread Neal, Phil 
 From: Dina Belova [mailto:dbel...@mirantis.com]
 On Friday, October 03, 2014 2:53 AM
 
 Igor,
 
 Personally this idea looks really nice to me, as this will help to avoid
 strange code being merged and not found via reviewing process.
 
 Cheers,
 Dina
 
 On Fri, Oct 3, 2014 at 12:40 PM, Igor Degtiarov
 idegtia...@mirantis.com wrote:
 Hi folks!
 
 I try too guess do we need in ceilometer checking new patches for
 critical errors with pylint?
 
 As far as I know Nova and Sahara and others have such check. Actually
 it is not checking of all project but comparing of the number of
 errors without new patch and with it, and if diff is more then 0 then
 patch are not taken.

Looking a bit deeper it seems that Nova struggled with false positives and 
resorted to https://review.openstack.org/#/c/28754/ , which layers some 
historical checking of git on top of pylint's tendency to check only the latest 
commit. I can't say I'm too deeply versed in the code,  but it's enough to make 
me wonder if we want to go that direction and avoid the issues altogether?

 
 I have taken as pattern Sahara's solution and proposed a patch for
 ceilometer:
 https://review.openstack.org/#/c/125906/
 
 Cheers,
 Igor Degtiarov
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 --
 Best regards,
 Dina Belova
 Software Engineer
 Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Treating notifications as a contract (CADF)

2014-09-04 Thread Neal, Phil 
 On 9/04/2014 Sandy Walsh wrote:
 Yesterday, we had a great conversation with Matt Rutkowski from IBM,
 one
 of the authors of the CADF spec.
 
 I was having a disconnect on what CADF offers and got it clarified.
 
 My assumption was CADF was a set of transformation/extraction rules
 for
 taking data from existing data structures and defining them as
 well-known things. For example, CADF needs to know who sent this
 notification. I thought CADF would give us a means to point at an
 existing data structure and say that's where you find it.
 
 But I was wrong. CADF is a full-on schema/data structure of its own. It
 would be a fork-lift replacement for our existing notifications.

This was my aha as well, following a similar discussion with Matt and team, 
but also note that they've articulated an approach for bolt-on changes that 
would enable CADF content in existing pipelines. 
(https://wiki.openstack.org/wiki/Ceilometer/blueprints/support-standard-audit-formats)

 However, if your service hasn't really adopted notifications yet (green
 field) or you can handle a fork-lift replacement, CADF is a good option.
 There are a few gotcha's though. If you have required data that is
 outside of the CADF spec, it would need to go in the attachment
 section of the notification and that still needs a separate schema to
 define it. Matt's team is very receptive to extending the spec to
 include these special cases though.
Agreed that Matt's team was very willing to extend, but I still wonder about 
having to migrate appended data from its pre-approval location to its 
permanent location, depending on the speed of the CADF standard update.
 
 Anyway, I've written up all the options (as I see them) [1] with the
 advantages/disadvantages of each approach. It's just a strawman, so
 bend/spindle/mutilate.
Cool...will add comments there.
 
 Look forward to feedback!
 -S
 
 
 [1] https://wiki.openstack.org/wiki/NotificationsAndCADF
 
 
 
 
 On 9/3/2014 12:30 PM, Sandy Walsh wrote:
  On 9/3/2014 11:32 AM, Chris Dent wrote:
  On Wed, 3 Sep 2014, Sandy Walsh wrote:
 
  We're chatting with IBM about CADF and getting down to specifics
 on
  their applicability to notifications. Once I get StackTach.v3 into
  production I'm keen to get started on revisiting the notification
  format and olso.messaging support for notifications.
 
  Perhaps a hangout for those keenly interested in doing something
 about this?
  That seems like a good idea. I'd like to be a part of that.
I would , too, and I would suggest that much of the Ceilometer team would 
  Unfortunately I won't be at summit but would like to contribute what
  I can before and after.
 
  I took some notes on this a few weeks ago and extracted what
 seemed
  to be the two main threads or ideas the were revealed by the
  conversation that happened in this thread:
 
   * At the micro level have versioned schema for notifications such
 that
 one end can declare I am sending version X of notification
 foo.bar.Y and the other end can effectively deal.
  Yes, that's table-stakes I think. Putting structure around the payload
  section.
 
  Beyond type and version we should be able to attach meta information
  like public/private visibility and perhaps hints for external mapping
  (this trait - that trait in CADF, for example).
 
   * At the macro level standardize a packaging or envelope of all
 notifications so that they can be consumed by very similar code.
 That is: constrain the notifications in some way so we can also
 constrain the consumer code.
  That's the intention of what we have now. The top level traits are
  standard, the payload is open. We really only require: message_id,
  timestamp and event_type. For auditing we need to cover Who, What,
 When,
  Where, Why, OnWhat, OnWhere, FromWhere.
 

To whit, I think we've made good progress in this by defining the what is the 
minimum content for PaaS service notifications and gotten agreement around  
https://review.openstack.org/#/c/113396/11/doc/source/format.rst
for the Juno release. It's been driven by many of these same questions but is 
fairly narrow in scope; it defines a minimum set of content, but doesn't tackle 
the question of structure (beyond trait typing). The timing seems right to dig 
deeper.

   These ideas serve two different purposes: One is to ensure that
   existing notification use cases are satisfied with robustness and
   provide a contract between two endpoints. The other is to allow a
   fecund notification environment that allows and enables many
   participants.
  Good goals. When Producer and Consumer know what to expect,
 things are
  good ... I know to find the Instance ID here. When the consumer
  wants to deal with a notification as a generic object, things get tricky
  (find the instance ID in the payload, What is the image type?, Is
  this an error notification?)
 
  Basically, how do we define the principle artifacts for

Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer

2014-04-23 Thread Neal, Phil 


 -Original Message-
 From: Ladislav Smola [mailto:lsm...@redhat.com]
 Sent: Wednesday, April 23, 2014 4:29 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer
 
 Hi Neal, thanks for response.
 
 So I would see it as UNDERCLOUD_USE_UI (TripleO UI can be placed only to
 Undercloud)
 
 And for overcloud: OVERCLOUD_USE_UI and
 OVERCLOUD_USE_CEILOMETER, cause in
 overcloud users might not want UI, but only data for billing. Does it
 sound reasonable?

Yep, agreed that UI/metering are different use cases. Will work 
OVERCLOUD_USE_CEILOMETER use case into changes first, then address UI later if 
someone else hasn't picked it up.

- Phil
 
 On 04/22/2014 06:23 PM, Neal, Phil wrote:
  From: Ladislav Smola [mailto:lsm...@redhat.com]
  Sent: Wednesday, April 16, 2014 8:37 AM
  To: openstack-dev@lists.openstack.org
  Subject: Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer
 
  No response so far, but -1 on the image element for making Ceilometer
  optional.
  Sorry for the delayed response, Ladislov. It turns out that the mailing list
 was filtering out these TripleO mails for me.
 
  Let me add a little context to that -1: given that a TripleO user may not 
  want
 to enable a UI layer at the undercloud level (there's a use case for using the
 undercloud solely for spinning up the overcloud), I think we want to support
 as small a footprint as possible.
 
  OK, so what about having variable in devtest_variables: USE_TRIPLEO_UI.
 
  I like this approach better...in fact I will look into adding something 
  similar
 into the changes I'm making to enable Ceilometer by default in the overcloud
 control node: https://review.openstack.org/#/c/89625/1
 
  It would add Undercloud Ceilometer, Tuskar-UI and Horizon. And
 Overcloud
  SNMPd.
 
  Defaulted to USE_TRIPLEO_UI=1 so we have UI stuff in CI.
 
  How does it sound?
 
  Perhaps specify something like UNDERCLOUD_USE_TRIPLEO_UI to be
 more specific on where this will be deployed.
  On 04/14/2014 01:31 PM, Ladislav Smola wrote:
  Hello,
 
  I am planning to add Ceilometer to Undercloud as default. Since
  Tuskar-UI uses
  it as primary source of metering samples and Tuskar should be in
  Undercloud
  as default, it made sense to me.
 
  So is my assumption correct or there are some reasons not to do this?
 
  Here are the reviews, that are adding working Undercloud Ceilometer:
  https://review.openstack.org/#/c/86915/
  https://review.openstack.org/#/c/86917/  (depends on the template
  change)
  https://review.openstack.org/#/c/87215/
 
  Configuration for automatic obtaining of stats from all Overcloud
  nodes via.
  SNMP will follow soon.
 
  Thanks,
  Ladislav
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer

2014-04-22 Thread Neal, Phil 
 From: Ladislav Smola [mailto:lsm...@redhat.com]
 Sent: Wednesday, April 16, 2014 8:37 AM
 To: openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [TripleO] [Tuskar] Undercloud Ceilometer
 
 No response so far, but -1 on the image element for making Ceilometer
 optional.

Sorry for the delayed response, Ladislov. It turns out that the mailing list 
was filtering out these TripleO mails for me.

Let me add a little context to that -1: given that a TripleO user may not want 
to enable a UI layer at the undercloud level (there's a use case for using the 
undercloud solely for spinning up the overcloud), I think we want to support as 
small a footprint as possible. 

 
 OK, so what about having variable in devtest_variables: USE_TRIPLEO_UI.
 

I like this approach better...in fact I will look into adding something similar 
into the changes I'm making to enable Ceilometer by default in the overcloud 
control node: https://review.openstack.org/#/c/89625/1

 It would add Undercloud Ceilometer, Tuskar-UI and Horizon. And Overcloud
 SNMPd.
 
 Defaulted to USE_TRIPLEO_UI=1 so we have UI stuff in CI.
 
 How does it sound?
 
Perhaps specify something like UNDERCLOUD_USE_TRIPLEO_UI to be more specific 
on where this will be deployed. 
 
 On 04/14/2014 01:31 PM, Ladislav Smola wrote:
  Hello,
 
  I am planning to add Ceilometer to Undercloud as default. Since
  Tuskar-UI uses
  it as primary source of metering samples and Tuskar should be in
  Undercloud
  as default, it made sense to me.
 
  So is my assumption correct or there are some reasons not to do this?
 
  Here are the reviews, that are adding working Undercloud Ceilometer:
  https://review.openstack.org/#/c/86915/
  https://review.openstack.org/#/c/86917/  (depends on the template
 change)
  https://review.openstack.org/#/c/87215/
 
  Configuration for automatic obtaining of stats from all Overcloud
  nodes via.
  SNMP will follow soon.
 
  Thanks,
  Ladislav
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] [QA] Slow Ceilometer resource_list CLI command

2014-03-18 Thread Neal, Phil 

 -Original Message-
 From: Tim Bell [mailto:tim.b...@cern.ch]
 Sent: Monday, March 17, 2014 2:04 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] [QA] Slow Ceilometer
 resource_list CLI command
 
 
 At CERN, we've had similar issues when enabling telemetry. Our resource-list
 times out after 10 minutes when the proxies for HA assume there is no
 answer coming back. Keystone instances per cell have helped the situation a
 little so we can collect the data but there was a significant increase in 
 load on
 the API endpoints.
 
 I feel that some reference for production scale validation would be beneficial
 as part of TC approval to leave incubation in case there are issues such as 
 this
 to be addressed.
 
 Tim
 
  -Original Message-
  From: Jay Pipes [mailto:jaypi...@gmail.com]
  Sent: 17 March 2014 20:25
  To: openstack-dev@lists.openstack.org
  Subject: Re: [openstack-dev] [Ceilometer] [QA] Slow Ceilometer
 resource_list CLI command
 
 ...
 
  Yep. At ATT, we had to disable calls to GET /resources without any filters
 on it. The call would return hundreds of thousands of
  records, all being JSON-ified at the Ceilometer API endpoint, and the result
 would take minutes to return. There was no default limit
  on the query, which meant every single records in the database was
 returned, and on even a semi-busy system, that meant
  horrendous performance.
 
  Besides the problem that the SQLAlchemy driver doesn't yet support
 pagination [1], the main problem with the get_resources() call is
  the underlying databases schema for the Sample model is wacky, and
 forces the use of a dependent subquery in the WHERE clause
  [2] which completely kills performance of the query to get resources.
 
  [1]
 
 https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/
 impl_sqlalchemy.py#L436
  [2]
 
 https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/
 impl_sqlalchemy.py#L503
 
   The cli tests are supposed to be quick read-only sanity checks of the
   cli functionality and really shouldn't ever be on the list of slowest
   tests for a gate run.
 
  Oh, the test is readonly all-right. ;) It's just that it's reading hundreds 
  of
 thousands of records.
 
I think there was possibly a performance regression recently in
   ceilometer because from I can tell this test used to normally take ~60 
   sec.
   (which honestly is probably too slow for a cli test too) but it is
   currently much slower than that.
  
   From logstash it seems there are still some cases when the resource
   list takes as long to execute as it used to, but the majority of runs 
   take a
 long time:
   http://goo.gl/smJPB9
  
   In the short term I've pushed out a patch that will remove this test
   from gate
   runs: https://review.openstack.org/#/c/81036 But, I thought it would
   be good to bring this up on the ML to try and figure out what changed
   or why this is so slow.
 
  I agree with removing the test from the gate in the short term. Medium to
 long term, the root causes of the problem (that GET
  /resources has no support for pagination on the query, there is no default
 for limiting results based on a since timestamp, and that
  the underlying database schema is non-optimal) should be addressed.

Gordon has introduced a blueprint 
https://blueprints.launchpad.net/ceilometer/+spec/big-data-sql with some fixes 
for individual queries but +1 to the point of looking at re-architecting the 
schema as an approach to fixing performance. We've also seen some gains here at 
HP using batch writes as well but have temporarily tabled that work in favor of 
getting a better-performing schema in place.
- Phil

 
  Best,
  -jay
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-07 Thread Neal, Phil 
For multi-node deployments, implementing something like inotify would allow 
administrators to push configuration changes out to multiple targets using 
puppet/chef/etc. and have the daemons pick it up without restart. Thumbs up to 
that.

As Tim Bell suggested, API-based enabling/disabling would allow users to update 
meters via script, but then there's the question of how to work out the global 
vs. per-project tenant settings...right now we collect specified meters for all 
available projects, and the API returns whatever data is stored minus filtered 
values. Maybe I'm missing something in the suggestion, but turning off 
collection for an individual project seems like it'd require some deep changes.

Vijay, I'll repeat dhellmann's request: do you have more detail in another doc? 
:-)

-   Phil

 -Original Message-
 From: Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
 [mailto:vijayakumar.kodam@nsn.com]
 Sent: Tuesday, January 07, 2014 2:49 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Cc: chmo...@enovance.com
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 From: ext Chmouel Boudjnah [mailto:chmo...@enovance.com]
 Sent: Monday, January 06, 2014 2:19 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 
 
 
 
 
 On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata Consultancy
 Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:
 
 In this case, simply changing the meter properties in a configuration file
 should be enough. There should be an inotify signal which shall notify
 ceilometer of the changes in the config file. Then ceilometer should
 automatically update the meters without restarting.
 
 
 
 Why it cannot be something configured by the admin with inotifywait(1)
 command?
 
 
 
 Or this can be an API call for enabling/disabling meters which could be more
 useful without having to change the config files.
 
 
 
 Chmouel.
 
 
 
 I haven't tried inotifywait() in this implementation. I need to check if it 
 will be
 useful for the current implementation.
 
 Yes. API call could be more useful than changing the config files manually.
 
 
 
 Thanks,
 
 VijayKumar

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Nomination of Sandy Walsh to core team

2013-12-16 Thread Neal, Phil 
+1. Sandy has been both helpful and insightful, plus he happens to have a good 
handle on the many moving parts that make up this project. :-)

-Phil

 -Original Message-
 From: Lu, Lianhao [mailto:lianhao...@intel.com]
 Sent: Sunday, December 15, 2013 5:29 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Nomination of Sandy Walsh to
 core team
 +1 in support.
 
 -Lianhao
 
 Nicolas Barcet wrote on 2013-12-13:
 +1 in support of Sandy.  He is a proven contributor and reviewer and he
 brings a great business vision and experience to the team.
 
 Cheers,
 Nick
 
 
 On Wed, Dec 11, 2013 at 8:18 PM, Gordon Chung chu...@ca.ibm.com wrote:
 
 
   To that end, I would like to nominate Sandy Walsh from Rackspace to
   ceilometer-core. Sandy is one of the original authors of StackTach,
 and   spearheaded the original stacktach-ceilometer integration. He
 has been  instrumental in many of my codes reviews, and has
 contributed much of the   existing event storage and querying code.
 
 
  +1 in support of Sandy.  the Event work he's led in Ceilometer has
 been an important feature and i think he has some valuable ideas.
 
  cheers, gordon chungopenstack, ibm software standards
  ___ OpenStack-dev 
 mailing
 list OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] [qa] Ceilometer ERRORS in normal runs

2013-10-21 Thread Neal, Phil 
Sean, we currently have a BP out there to investigate basic tempest
integration and I think this might fall under the same umbrella. 
Unfortunately I've not been able to free up my development time 
for it, but I've assigned it out to someone who can take a look and 
report back.

https://blueprints.launchpad.net/tempest/+spec/basic-tempest-integration-for-ceilometer

- Phil

 -Original Message-
 From: Sean Dague [mailto:s...@dague.net]
 Sent: Sunday, October 20, 2013 7:39 AM
 To: OpenStack Development Mailing List
 Subject: [openstack-dev] [ceilometer] [qa] Ceilometer ERRORS in normal
 runs
 
 Dave Kranz has been building a system so that we can ensure that during
 a Tempest run services don't spew ERRORs in the logs. Eventually, we're
 going to gate on this, because there is nothing that Tempest does to the
 system that should cause any OpenStack service to ERROR or stack trace
 (Errors should actually be exceptional events that something is wrong
 with the system, not regular events).
 
 Ceilometer is currently one of the largest offenders in dumping ERRORs
 in the gate -
 http://logs.openstack.org/68/52768/1/check/check-tempest-devstack-vm-
 full/76f83a4/console.html#_2013-10-19_14_51_51_271
 (that item isn't in our whitelist yet, so you'll see a lot of it at the
 end of every run)
 
 and
 http://logs.openstack.org/68/52768/1/check/check-tempest-devstack-vm-
 full/76f83a4/logs/screen-ceilometer-collector.txt.gz?level=TRACE
 for full details
 
 This seems like something is wrong in the integration, and would be
 really helpful if we could get ceilometer eyes on this one to put ceilo
 into a non erroring state.
 
   -Sean
 
 --
 Sean Dague
 http://dague.net
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Notifications from non-local exchanges

2013-10-15 Thread Neal, Phil 


 -Original Message-
 From: Sandy Walsh [mailto:sandy.wa...@rackspace.com]
 Sent: Thursday, October 10, 2013 6:20 PM
 To: openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Ceilometer] Notifications from non-local
 exchanges
 
 
 On 10/10/2013 06:16 PM, Neal, Phil wrote:
 Greetings all, I'm looking at how to expand the ability of our CM
 instance to consume notifications and have a quick question about
 the configuration and flow...
 
 For the notifications central agent ,  we rely on the services (i.e.
 glance, cinder)  to drop messages on the same messaging host as used
 by Ceilometer. From there the listener picks it up and cycles through
 the plugin logic to convert it to a sample. It's apparent that we
 can't pass an alternate hostname via the control_exchange values, so
 is there another method for harvesting messages off of other
 instances (e.g. another compute node)?
 
 Hey Phil,
 
 You don't really need to specify the exchange name to consume
 notifications. It will default to the control-exchange if not specified
 anyway.
 
 How it works isn't so obvious.
 
 Depending on the priority of then notification the oslo notifier will
 publish on topic.priority using the service's control-exchange. If
 that queue doesn't exist it'll create it and bind the control-exchange
 to it. This is so we can publish even if there are no consumers yet.

I think the common default is notifications.info, yes?

 
 Oslo.rpc creates a 1:1 mapping of routing_key and queue to topic (no
 wildcards). So we get
 
 exchange:service - binding: routing_key topic.priority -
 queue topic.priority
 
 (essentially, 1 queue per priority)
 
 Which is why, if you want to enable services to generate notifications,
 you just have to set the driver and the topic(s) to publish on. Exchange
 is implied and routing key/queue are inferred from topic.

Yep, following up to this point: Oslo takes care of the setup of exchanges on 
behalf of the 
services. When, say, Glance wants to push notifications onto the message bus, 
they can set 
the control_exchange value and the driver (rabbit, for example) and voila! An 
exchange is
set up with a default queue bound to the key. 
 
 Likewise we only have to specify the queue name to consume, since we
 only need an exchange to publish.

Here's where my gap is: the notification plugins seem to assume that Ceilometer 
is sitting on the same messaging node/endpoint as the service. The config file 
allows
us to specify the exchange names for the services , but not endpoints, so if 
Glance 
is publishing to notifications.info on rabbit.glance.hpcloud.net, and 
ceilometer
 is  publishing/consuming from the rabbit.ceil.hpcloud.net node then the 
Glance
 notifications won't be collected.

 I took another look at the Ceilometer config options...rabbit_hosts
takes multiple hosts (i.e. rabbit.glance.hpcloud.net:, 
rabbit.ceil.hpcloud.net:) 
but it's not clear whether that's for publishing, collection, or both?  The 
impl_kombu
module does cycle through that list to create the connection pool, but it's not
clear to me how it all comes together in the plugin instantiation...

 
 I have a bare-bones oslo notifier consumer and client here if you want
 to mess around with it (and a bare-bones kombu version in the parent).

Will take a look! 

 
 https://github.com/SandyWalsh/amqp_sandbox/tree/master/oslo
 
 Not sure if that answered your question or made it worse? :)
 
 Cheers
 -S
 
 
 
 
 - Phil
 
 ___ OpenStack- dev mailing
 list OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Notifications from non-local exchanges

2013-10-10 Thread Neal, Phil 
Greetings all,
I'm looking at how to expand the ability of our CM instance to consume 
notifications and have a quick question about  the configuration and flow...

For the notifications central agent ,  we rely on the services (i.e. glance, 
cinder)  to drop messages on the same messaging host as used by Ceilometer. 
From there the listener picks it up and cycles through the plugin logic to 
convert it to a sample. It's apparent that we can't pass an alternate hostname 
via the control_exchange values, so is there another method for harvesting 
messages off of other  instances (e.g. another compute node)?


- Phil

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Ceilometer] Correct way to disable specific event collection by the collector

2013-09-11 Thread Neal, Phil 
Greetings team,
I'm working on getting a very streamlined set of collections running and I'd 
like to disable all notifications except Glance. It's clear that the desired 
event types are defined in the plugins, but I can't seem to work out how to 
force the collector service to load only specific handlers in the 
ceilometer.collector namespace. I *thought* it could be accomplished by 
editing /ceilometer/setup.cfg, but removing the entry points there didn't seem 
to work (the extensions manager still picks them up).

Can someone give me a rough idea of how to do this?

- Phil 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Changing pollster/pipeline parameters

2013-07-08 Thread Neal, Phil 
 -Original Message-
 From: Eoghan Glynn [mailto:egl...@redhat.com]
 Sent: Thursday, July 04, 2013 3:36 AM
 To: OpenStack Development Mailing List; Neal, Phil
 Subject: Re: [openstack-dev] [Ceilometer] Changing pollster/pipeline
 parameters
 
 
 Hi Phil,
 
 Some more thoughts inline ...
 
   A couple of related questions on managing CM pollster behavior via
 config
   type files:
  
   1. I'm trying to make some modifications to the timing of the Glance
   (image)
   polling in a default CM install. It looks like the pipeline interval 
   fields
   are the way to do it, and that I should tweak it using a yaml config file,
   but I can't seem to verify based on a read-through of the code. Can
 anyone
   confirm?
 
  Yep, the configured interval in the /etc/ceilometer/pipeline.yaml is the way
  to go.
 
  For example:
 
   $ sed -i 's/interval: .*$/interval: 10/' /etc/ceilometer/pipeline.yaml
   $ # restart ceilometer-agent-central
   $ ceilometer sample-list -m image.size | tail -5
   | df52ee46-91c0-417a-a27c-ca0b447fb5db | image.size | gauge |
 25165824.0 | B
   | | 2013-07-03T21:40:15|
   | df52ee46-91c0-417a-a27c-ca0b447fb5db | image.size | gauge |
 25165824.0 | B
   | | 2013-07-03T21:40:25|
   | df52ee46-91c0-417a-a27c-ca0b447fb5db | image.size | gauge |
 25165824.0 | B
   | | 2013-07-03T21:40:35|
   | df52ee46-91c0-417a-a27c-ca0b447fb5db | image.size | gauge |
 25165824.0 | B
   | | 2013-07-03T21:40:45|
   
  +--++---++--+--
 --+
 
  As you see the meter is now being collected at a 10s cadence.
 
  This timing is pulled in via the AgentManager base class:
 
 
 https://github.com/openstack/ceilometer/blob/master/ceilometer/agent.p
 y#L90
 
 I should have been more explicit here, in the sense of pointing out that:
 
 (a) this pipeline config can be made specific to the central agent
 (i.e. the agent responsible for polling glance among other things)
 by adding the line:
 
   pipeline_cfg_file = pipeline-central.yaml
 
 to the ceilometer.conf picked up by the central agent (so that the
 central agent pipeline is independent of the pipeline used by the
 other agents)
 
 
 (b) multiple pipelines can be defined in a single yaml file, with different
 polling intervals for different groups of meters.
 
 For example, to make only the glance polling interval 120s while keeping
 the other central agent polling cycles at 60s, use something like the
 following in the appropriate pipeline.yaml file:
 
 ---
 -
 name: meter_pipeline
 interval: 60
 counters:
 - *
 - !image
 - !image.size
 transformers:
 publishers:
 - rpc://
 -
 name: image_pipeline
 interval: 120
 counters:
 - image
 - image.size
 transformers:
 publishers:
 - rpc://
 

Makes sense on paper. :=)  Will try implementing today.

 
   2. Similarly, it looks like disabling pollsters is done via the oslo.cfg
   logic in the agent manager. I'd like to populate that using a config
   fileis there logic already to do that that I haven't come across yet?
 
  Well the way I'd disable a pollster is simply by configuring the counters
  exclusion in the pipeline.yaml.
 
  For example to disable the pollster providing the image.size meter referred
  to above, the following will do the trick:
 
  counters:
  - *
  - !image.size
 
  i.e. allow everything but the image.size meter.
 
  Confirm as before using:
 
   $ # restart ceilometer-agent-central
   $ ceilometer sample-list -m image.size | tail -5
 
 
 It also occurred to me that you may have been thinking of the now-obsolete
 disabled_{central|compute|...}_pollsters config options, which are no longer
 supported as they've been superceeded by the newer pipeline config.
 

Yes, that's the line of logic I was headed down. Thanks for noting the 
obsolescence.

 Hope this helps clarify things for you.
 
 Cheers,
 Eoghan
 
 


Truly helpfulthanks again!
   - Phil
  
   ___
   OpenStack-dev mailing list
   OpenStack-dev@lists.openstack.org
   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
  
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Ceilometer] Changing pollster/pipeline parameters

2013-07-03 Thread Neal, Phil 
A couple of related questions on managing CM pollster behavior via config 
type files:

1. I'm trying to make some modifications to the timing of the Glance (image) 
polling in a default CM install. It looks like the pipeline interval fields are 
the way to do it, and that I should tweak it using a yaml config file, but I 
can't seem to verify based on a read-through of the code. Can anyone confirm?

2. Similarly, it looks like disabling pollsters is done via the oslo.cfg logic 
in the agent manager. I'd like to populate that using a config fileis there 
logic already to do that that I haven't come across yet?

- Phil

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev