Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-09 Thread Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)


From: ext Doug Hellmann [mailto:doug.hellm...@dreamhost.com]
Sent: Wednesday, January 08, 2014 10:21 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer



On Wed, Jan 8, 2014 at 3:09 PM, Kodam, Vijayakumar (EXT-Tata Consultancy Ser - 
FI/Espoo) vijayakumar.kodam@nsn.commailto:vijayakumar.kodam@nsn.com 
wrote:

 
 
 From: ext Doug Hellmann 
 [doug.hellm...@dreamhost.commailto:doug.hellm...@dreamhost.com]
 Sent: Wednesday, January 08, 2014 8:26 PM

 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 
 
 On Wed, Jan 8, 2014 at 12:35 PM, Ildikó Váncsa 
 ildiko.van...@ericsson.commailto:ildiko.van...@ericsson.com wrote:
 
 Hi Doug,
 
 Answers inline again.
 
 Best Regards,
 
 Ildiko
 
 
 On Wed, Jan 8, 2014 at 3:16 AM, Ildikó Váncsa 
 ildiko.van...@ericsson.commailto:ildiko.van...@ericsson.com wrote:
 
 Hi,
 
 I've started to work on the idea of supporting a kind of tenant/project
  based configuration for Ceilometer. Unfortunately I haven't reached
  the point of having a blueprint that could be registered until now.
  I do not have a deep knowledge about the collector and compute agent
  services, but this feature would require some deep changes for sure.
  Currently there are pipelines for data collection and transformation,
  where the counters can be specified, about which data should be
  collected and also the time interval for data collection and so on.
  These pipelines can be configured now globally in the pipeline.yaml file,
  which is stored right next to the Ceilometer configuration files.
 
 Yes, the data collection was designed to be configured and controlled by
  the deployer, not the tenant. What benefits do we gain by giving that
  control to the tenant?
 
 ildikov: Sorry, my explanation was not clear. I meant there the configuration
  of data collection for projects, what was mentioned by Tim Bell in a
  previous email. This would mean that the project administrator is able to
  create a data collection configuration for his/her own project, which will
  not affect the other project's configuration. The tenant would be able to
  specify meters (enabled/disable based on which ones are needed) for the given
  project also with project specific time intervals, etc.
 
 OK, I think some of the confusion is terminology.
 Who is a project administrator? Is that someone with access to change
  ceilometer's configuration file directly? Someone with a particular role
  using the API? Or something else?
 
 ildikov: As project administrator I meant a user with particular role,
  a user assigned to a tenant.
 
 
 OK, so like I said, we did not design the system with the idea that a
  user of the cloud (rather than the deployer of the cloud) would have
  any control over what data was collected. They can ask questions about
  only some of the data, but they can't tell ceilometer what to collect.
 There's a certain amount of danger in giving the cloud user
  (no matter their role) an off switch for the data collection.
 
  As Julien pointed out, it can have a negative effect on billing
  -- if they tell the cloud not to collect data about what instances
  are created, then the deployer can't bill for those instances.
  Differentiating between the values that always must be collected and
  the ones the user can control makes providing an API to manage data
  collection more complex.
 
 Is there some underlying use case behind all of this that someone could
  describe in more detail, so we might be able to find an alternative, or
  explain how to use the existing features to achieve the goal?
 
  For example, it is already possible to change the pipeline config file
  to control which data is collected and stored.
  If we make the pipeline code in ceilometer watch for changes to that file,
  and rebuild the pipelines when the config is updated,
  would that satisfy the requirements?
 

Yes. That's exactly the requirement for our blueprint. To avoid ceilometer 
restart for changes to take effect, when the config file changes.
API support was added later based on the request in this mail chain. We 
actually don't need APIs and can be removed.

So as you mentioned above, whenever the config file is changed, ceilometer 
should update the meters accordingly.

OK, I think that's something reasonable to implement, although I would have to 
look at the collector to make sure we could rebuild the pipelines safely 
without losing any data as more messages come in. But it should be possible, if 
not easy. :-)

The blueprint should be updated to reflect this approach.

Doug

Thanks Doug.
I shall update the blueprint accordingly.

VijayKumar


 
 
 In my view, we could keep the dynamic meter configuration bp with considering
  to extend it to dynamic

Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Ildikó Váncsa
Hi,

I've started to work on the idea of supporting a kind of tenant/project based 
configuration for Ceilometer. Unfortunately I haven't reached the point of 
having a blueprint that could be registered until now. I do not have a deep 
knowledge about the collector and compute agent services, but this feature 
would require some deep changes for sure. Currently there are pipelines for 
data collection and transformation, where the counters can be specified, about 
which data should be collected and also the time interval for data collection 
and so on. These pipelines can be configured now globally in the pipeline.yaml 
file, which is stored right next to the Ceilometer configuration files.

In my view, we could keep the dynamic meter configuration bp with considering 
to extend it to dynamic configuration of Ceilometer, not just the meters and we 
could have a separate bp for the project based configuration of meters.

If it is ok for you, I will register the bp for this per-project tenant 
settings with some details, when I'm finished with the initial design of how 
this feature could work.

Best Regards,
Ildiko

-Original Message-
From: Neal, Phil [mailto:phil.n...@hp.com] 
Sent: Tuesday, January 07, 2014 11:50 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

For multi-node deployments, implementing something like inotify would allow 
administrators to push configuration changes out to multiple targets using 
puppet/chef/etc. and have the daemons pick it up without restart. Thumbs up to 
that.

As Tim Bell suggested, API-based enabling/disabling would allow users to update 
meters via script, but then there's the question of how to work out the global 
vs. per-project tenant settings...right now we collect specified meters for all 
available projects, and the API returns whatever data is stored minus filtered 
values. Maybe I'm missing something in the suggestion, but turning off 
collection for an individual project seems like it'd require some deep changes.

Vijay, I'll repeat dhellmann's request: do you have more detail in another doc? 
:-)

-   Phil

 -Original Message-
 From: Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo) 
 [mailto:vijayakumar.kodam@nsn.com]
 Sent: Tuesday, January 07, 2014 2:49 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Cc: chmo...@enovance.com
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 From: ext Chmouel Boudjnah [mailto:chmo...@enovance.com]
 Sent: Monday, January 06, 2014 2:19 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 
 
 
 
 
 On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata 
 Consultancy Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:
 
 In this case, simply changing the meter properties in a configuration 
 file should be enough. There should be an inotify signal which shall 
 notify ceilometer of the changes in the config file. Then ceilometer 
 should automatically update the meters without restarting.
 
 
 
 Why it cannot be something configured by the admin with inotifywait(1) 
 command?
 
 
 
 Or this can be an API call for enabling/disabling meters which could 
 be more useful without having to change the config files.
 
 
 
 Chmouel.
 
 
 
 I haven't tried inotifywait() in this implementation. I need to check 
 if it will be useful for the current implementation.
 
 Yes. API call could be more useful than changing the config files manually.
 
 
 
 Thanks,
 
 VijayKumar

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Doug Hellmann
On Wed, Jan 8, 2014 at 3:16 AM, Ildikó Váncsa ildiko.van...@ericsson.comwrote:

 Hi,

 I've started to work on the idea of supporting a kind of tenant/project
 based configuration for Ceilometer. Unfortunately I haven't reached the
 point of having a blueprint that could be registered until now. I do not
 have a deep knowledge about the collector and compute agent services, but
 this feature would require some deep changes for sure. Currently there are
 pipelines for data collection and transformation, where the counters can be
 specified, about which data should be collected and also the time interval
 for data collection and so on. These pipelines can be configured now
 globally in the pipeline.yaml file, which is stored right next to the
 Ceilometer configuration files.


Yes, the data collection was designed to be configured and controlled by
the deployer, not the tenant. What benefits do we gain by giving that
control to the tenant?



 In my view, we could keep the dynamic meter configuration bp with
 considering to extend it to dynamic configuration of Ceilometer, not just
 the meters and we could have a separate bp for the project based
 configuration of meters.


Ceilometer uses oslo.config, just like all of the rest of OpenStack. How
are the needs for dynamic configuration updates in ceilometer different
from the other services?

Doug




 If it is ok for you, I will register the bp for this per-project tenant
 settings with some details, when I'm finished with the initial design of
 how this feature could work.

 Best Regards,
 Ildiko

 -Original Message-
 From: Neal, Phil [mailto:phil.n...@hp.com]
 Sent: Tuesday, January 07, 2014 11:50 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

 For multi-node deployments, implementing something like inotify would
 allow administrators to push configuration changes out to multiple targets
 using puppet/chef/etc. and have the daemons pick it up without restart.
 Thumbs up to that.

 As Tim Bell suggested, API-based enabling/disabling would allow users to
 update meters via script, but then there's the question of how to work out
 the global vs. per-project tenant settings...right now we collect specified
 meters for all available projects, and the API returns whatever data is
 stored minus filtered values. Maybe I'm missing something in the
 suggestion, but turning off collection for an individual project seems like
 it'd require some deep changes.

 Vijay, I'll repeat dhellmann's request: do you have more detail in another
 doc? :-)

 -   Phil

  -Original Message-
  From: Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
  [mailto:vijayakumar.kodam@nsn.com]
  Sent: Tuesday, January 07, 2014 2:49 AM
  To: OpenStack Development Mailing List (not for usage questions)
  Cc: chmo...@enovance.com
  Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
  From: ext Chmouel Boudjnah [mailto:chmo...@enovance.com]
  Sent: Monday, January 06, 2014 2:19 PM
  To: OpenStack Development Mailing List (not for usage questions)
  Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 
 
 
 
 
  On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata
  Consultancy Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:
 
  In this case, simply changing the meter properties in a configuration
  file should be enough. There should be an inotify signal which shall
  notify ceilometer of the changes in the config file. Then ceilometer
  should automatically update the meters without restarting.
 
 
 
  Why it cannot be something configured by the admin with inotifywait(1)
  command?
 
 
 
  Or this can be an API call for enabling/disabling meters which could
  be more useful without having to change the config files.
 
 
 
  Chmouel.
 
 
 
  I haven't tried inotifywait() in this implementation. I need to check
  if it will be useful for the current implementation.
 
  Yes. API call could be more useful than changing the config files
 manually.
 
 
 
  Thanks,
 
  VijayKumar

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
Hi,

-Original Message-
From: ext Neal, Phil [mailto:phil.n...@hp.com] 
Sent: Wednesday, January 08, 2014 12:50 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer


For multi-node deployments, implementing something like inotify would allow 
administrators to push configuration changes out to multiple targets using 
puppet/chef/etc. and have the daemons pick it up without restart. Thumbs up 
to that.


Thanks!

As Tim Bell suggested, API-based enabling/disabling would allow users to 
update meters via script, but then there's the question of how to work out the 
global vs. per-project tenant settings...right now we collect specified 
meters for all available projects, and the API returns whatever data is stored 
minus filtered values. Maybe I'm missing something in the suggestion, but 
turning off collection for an individual project seems like it'd require some 
deep changes.

Vijay, I'll repeat dhellmann's request: do you have more detail in another 
doc? :-)

-  Phil

I concur with the opinion to use APIs for dynamically enabling/disabling 
meters. I have updated the design accordingly.

According to the latest update:
User calls the API(1) to disable a meter along with a meter id. 
Ceilometer-api handles the api request, adds the meter id to disabled_meters 
config file and informs ceilometer agents. 
Ceilometer agents will read the disabled_meters config file and disables the 
meter.

More detailed information about this blueprint can be found at
https://etherpad.openstack.org/p/dynamic-meters

There will be no inotify() or inotifywait() calls to monitor the modifications 
of the configuration file.
Whenever the APIs are called, ceilometer-api upon receiving the request, will 
modify the config file and informs the ceilometer agents.

There are no per-project settings currently considered for this blueprint. 
IMHO, per-project settings should be implemented for all the 
meters/resources/APIs in ceilometer and should be handled by a different 
blueprint.

Regards,
VijayKumar

 -Original Message-
 From: Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
 [mailto:vijayakumar.kodam@nsn.com]
 Sent: Tuesday, January 07, 2014 2:49 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Cc: chmo...@enovance.com
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 From: ext Chmouel Boudjnah [mailto:chmo...@enovance.com]
 Sent: Monday, January 06, 2014 2:19 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 
 
 
 
 
 On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata Consultancy
 Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:
 
 In this case, simply changing the meter properties in a configuration file
 should be enough. There should be an inotify signal which shall notify
 ceilometer of the changes in the config file. Then ceilometer should
 automatically update the meters without restarting.
 
 
 
 Why it cannot be something configured by the admin with inotifywait(1)
 command?
 
 
 
 Or this can be an API call for enabling/disabling meters which could be more
 useful without having to change the config files.
 
 
 
 Chmouel.
 
 
 
 I haven't tried inotifywait() in this implementation. I need to check if it 
 will be
 useful for the current implementation.
 
 Yes. API call could be more useful than changing the config files manually.
 
 
 
 Thanks,
 
 VijayKumar

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)


From: ext Tim Bell [mailto:tim.b...@cern.ch]
Sent: Tuesday, January 07, 2014 8:22 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer


Thinking using inotify/configuration file changes to implement dynamic meters, 
this would be limited to administrators of ceilometer itself (i.e. with write 
access to the file) rather than the project administrators (as defined by 
keystone roles). Thus, as a project administrator who is not the cloud admin, I 
could not enable/disable the meter for a project only.

It would mean that scripting meter on/off would not be possible if there was 
not an API to perform this.

Not sure if these requirements are significant and the associated impact on 
implementation complexity, but they may be relevant in scoping out the 
blueprint and subsequent changes

Tim
Tim,

Agree with your suggestion. I have updated the design by adding APIs. Whenever 
an API request is received by the ceilometer-api, it shall modify the config 
file and inform the ceilometer agents.
You can find detailed information at
https://etherpad.openstack.org/p/dynamic-meters

Regards,
VijayKumar

From: Doug Hellmann [mailto:doug.hellm...@dreamhost.com]
Sent: 06 January 2014 23:35
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer



On Tue, Dec 31, 2013 at 4:53 AM, Kodam, Vijayakumar (EXT-Tata Consultancy Ser - 
FI/Espoo) vijayakumar.kodam@nsn.commailto:vijayakumar.kodam@nsn.com 
wrote:
Hi,

Currently there is no way to enable or disable meters without restarting 
ceilometer.

There are cases where operators do not want to run all the meters continuously.
In these cases, there should be a way to disable or enable them dynamically.

We are working on this feature right now. I have also created a blueprint for 
the same.
https://blueprints.launchpad.net/ceilometer/+spec/dynamic-meters

We would love to hear your views on this feature.

There isn't much detail in the blueprint. Do you have a more comprehensive 
document you can link to that talks about how you intend for it to work?

Doug



Regards,
VijayKumar Kodam




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Julien Danjou
On Wed, Jan 08 2014, Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo) 
wrote:

 According to the latest update:
 User calls the API(1) to disable a meter along with a meter id. 

What's an user? An end-user or an operator?
I don't think we want to allow a user to disable a meter. I don't see
the use case, and if you use a meter for billing, it's just a terrible
idea.

If you talk about operators, it's just a problem of managing a
configuration file that is not different than the rest of OpenStack. I
think Doug and Chmouel already answered that, and I sit on that line.

-- 
Julien Danjou
;; Free Software hacker ; independent consultant
;; http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Julien Danjou
On Wed, Jan 08 2014, Ildikó Váncsa wrote:

(Your answers are very hard to read inline in my text MUA, it'd really
help if you could quote properly with  the emails you answer to).

 ildikov: Sorry, my explanation was not clear. I meant there the
 configuration of data collection for projects, what was mentioned by Tim
 Bell in a previous email. This would mean that the project administrator is
 able to create a data collection configuration for his/her own project,
 which will not affect the other project's configuration. The tenant would be
 able to specify meters (enabled/disable based on which ones are needed) for
 the given project also with project specific time intervals, etc.

I still don't see the point. A user can send any sample it wants on any
interval using the REST API. There's no sense of enable or disabling
meters.
Please describe me a real use case, for now I still can't understand
what you want to do.

-- 
Julien Danjou
/* Free Software hacker * independent consultant
   http://julien.danjou.info */


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Doug Hellmann
On Wed, Jan 8, 2014 at 11:16 AM, Ildikó Váncsa
ildiko.van...@ericsson.comwrote:

  Hi Doug,



 See my answers inline.



 Best Regards,

 Ildiko



 *From:* Doug Hellmann [mailto:doug.hellm...@dreamhost.com]
 *Sent:* Wednesday, January 08, 2014 4:10 PM

 *To:* OpenStack Development Mailing List (not for usage questions)
 *Subject:* Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer







 On Wed, Jan 8, 2014 at 3:16 AM, Ildikó Váncsa ildiko.van...@ericsson.com
 wrote:

 Hi,

 I've started to work on the idea of supporting a kind of tenant/project
 based configuration for Ceilometer. Unfortunately I haven't reached the
 point of having a blueprint that could be registered until now. I do not
 have a deep knowledge about the collector and compute agent services, but
 this feature would require some deep changes for sure. Currently there are
 pipelines for data collection and transformation, where the counters can be
 specified, about which data should be collected and also the time interval
 for data collection and so on. These pipelines can be configured now
 globally in the pipeline.yaml file, which is stored right next to the
 Ceilometer configuration files.



 Yes, the data collection was designed to be configured and controlled by
 the deployer, not the tenant. What benefits do we gain by giving that
 control to the tenant?



 ildikov: Sorry, my explanation was not clear. I meant there the
 configuration of data collection for projects, what was mentioned by Tim
 Bell in a previous email. This would mean that the project administrator is
 able to create a data collection configuration for his/her own project,
 which will not affect the other project’s configuration. The tenant would
 be able to specify meters (enabled/disable based on which ones are needed)
 for the given project also with project specific time intervals, etc.


OK, I think some of the confusion is terminology. Who is a project
administrator? Is that someone with access to change ceilometer's
configuration file directly? Someone with a particular role using the API?
Or something else?






 In my view, we could keep the dynamic meter configuration bp with
 considering to extend it to dynamic configuration of Ceilometer, not just
 the meters and we could have a separate bp for the project based
 configuration of meters.



 Ceilometer uses oslo.config, just like all of the rest of OpenStack. How
 are the needs for dynamic configuration updates in ceilometer different
 from the other services?



 ildikov: There are some parameters in the configuration file of
 Ceilometer, like log options and notification types, which would be good to
 be able to configure them dynamically. I just wanted to reflect to that
 need. As I see, there are two options here. The first one is to identify
 the group of the dynamically modifiable parameters and move them to the API
 level. The other option could be to make some modifications in oslo.config
 too, so other services also could use the benefits of dynamic
 configuration. For example the log settings could be a good candidate, as
 for example the change of log levels, without service restart, in case
 debugging the system can be a useful feature for all of the OpenStack
 services.


I misspoke earlier. If we're talking about meters, those are actually
defined by the pipeline file (not oslo.config). So if we do want that file
re-read automatically, we can implement that within ceilometer itself,
though I'm still reluctant to say we want to provide API access for
modifying those settings. That's *really* not something we've designed the
rest of the system to accommodate, so I don't know what side-effects we
might introduce.

As far as the other configuration settings, we had the conversation about
updating those through some sort of API early on, and decided that there
are already lots of operational tools out there to manage changes to those
files. I would need to see a list of which options people would want to
have changed through an API to comment further.

Doug





 Doug






 If it is ok for you, I will register the bp for this per-project tenant
 settings with some details, when I'm finished with the initial design of
 how this feature could work.

 Best Regards,
 Ildiko


 -Original Message-
 From: Neal, Phil [mailto:phil.n...@hp.com]
 Sent: Tuesday, January 07, 2014 11:50 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

 For multi-node deployments, implementing something like inotify would
 allow administrators to push configuration changes out to multiple targets
 using puppet/chef/etc. and have the daemons pick it up without restart.
 Thumbs up to that.

 As Tim Bell suggested, API-based enabling/disabling would allow users to
 update meters via script, but then there's the question of how to work out
 the global vs. per-project tenant settings...right now we collect specified

Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Ildikó Váncsa
Hi Doug,

Answers inline again.

Best Regards,
Ildiko

On Wed, Jan 8, 2014 at 3:16 AM, Ildikó Váncsa 
ildiko.van...@ericsson.commailto:ildiko.van...@ericsson.com wrote:
Hi,

I've started to work on the idea of supporting a kind of tenant/project based 
configuration for Ceilometer. Unfortunately I haven't reached the point of 
having a blueprint that could be registered until now. I do not have a deep 
knowledge about the collector and compute agent services, but this feature 
would require some deep changes for sure. Currently there are pipelines for 
data collection and transformation, where the counters can be specified, about 
which data should be collected and also the time interval for data collection 
and so on. These pipelines can be configured now globally in the pipeline.yaml 
file, which is stored right next to the Ceilometer configuration files.

Yes, the data collection was designed to be configured and controlled by the 
deployer, not the tenant. What benefits do we gain by giving that control to 
the tenant?

ildikov: Sorry, my explanation was not clear. I meant there the configuration 
of data collection for projects, what was mentioned by Tim Bell in a previous 
email. This would mean that the project administrator is able to create a data 
collection configuration for his/her own project, which will not affect the 
other project's configuration. The tenant would be able to specify meters 
(enabled/disable based on which ones are needed) for the given project also 
with project specific time intervals, etc.

OK, I think some of the confusion is terminology. Who is a project 
administrator? Is that someone with access to change ceilometer's 
configuration file directly? Someone with a particular role using the API? Or 
something else?

ildikov: As project administrator I meant a user with particular role, a user 
assigned to a tenant.




In my view, we could keep the dynamic meter configuration bp with considering 
to extend it to dynamic configuration of Ceilometer, not just the meters and we 
could have a separate bp for the project based configuration of meters.

Ceilometer uses oslo.config, just like all of the rest of OpenStack. How are 
the needs for dynamic configuration updates in ceilometer different from the 
other services?

ildikov: There are some parameters in the configuration file of Ceilometer, 
like log options and notification types, which would be good to be able to 
configure them dynamically. I just wanted to reflect to that need. As I see, 
there are two options here. The first one is to identify the group of the 
dynamically modifiable parameters and move them to the API level. The other 
option could be to make some modifications in oslo.config too, so other 
services also could use the benefits of dynamic configuration. For example the 
log settings could be a good candidate, as for example the change of log 
levels, without service restart, in case debugging the system can be a useful 
feature for all of the OpenStack services.

I misspoke earlier. If we're talking about meters, those are actually defined 
by the pipeline file (not oslo.config). So if we do want that file re-read 
automatically, we can implement that within ceilometer itself, though I'm still 
reluctant to say we want to provide API access for modifying those settings. 
That's *really* not something we've designed the rest of the system to 
accommodate, so I don't know what side-effects we might introduce.

ildikov: In case of oslo.config, I meant the ceilometer.conf file in my answer 
above, not pipeline.yaml. As for the API part, I do not know the consequences 
of that implementation either, so now I'm kind of waiting for the outcome of 
this Dynamic Meters bp.

As far as the other configuration settings, we had the conversation about 
updating those through some sort of API early on, and decided that there are 
already lots of operational tools out there to manage changes to those files. I 
would need to see a list of which options people would want to have changed 
through an API to comment further.

ildikov: Yes, I agree that not all the parameters should be configured 
dynamically. It just popped into my mind regarding the dynamic configuration, 
that there would be a need to configure other configuration parameters, not 
just meters, that is why I mentioned it as a considerable item.

Doug



Doug



If it is ok for you, I will register the bp for this per-project tenant 
settings with some details, when I'm finished with the initial design of how 
this feature could work.

Best Regards,
Ildiko

-Original Message-
From: Neal, Phil [mailto:phil.n...@hp.commailto:phil.n...@hp.com]
Sent: Tuesday, January 07, 2014 11:50 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

For multi-node deployments, implementing something like inotify would allow 
administrators to push configuration changes out

Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Ildikó Váncsa
Hi,

(You didn't Cc the list, not sure if it was on purpose. I'm not adding it back 
to not break any confidentiality, but feel free to do so.)

Sorry that was just a mistake.

  The point is to configure the data collection configuration for the 
  currently existing meters differently for tenants. It is not just 
  about enabling or disabling of meters. It could be used to change the 
  interval settings of meters, like tenantA would like to receive 
  cpu_util samples in every 10 seconds and tenantB would like to receive 
  cpu_util in every 1 minute, but network.incoming.bytes in every 20 
  seconds. As for disabling meters, for instance tenantA needs 
  disk.read.requests and disk.write.requests, while tenantB doesn't.

 Ok, so this is really about something the _operator_ wants to do, not users. 
 I still don't think it belongs to an API, at least not specific to Ceilometer.

My idea was just about providing the possibility to configure the data 
collection in Ceilometer differently for the different tenants, I didn't mean 
to link it to an API or at least not on the first place. It could be done by 
the operator as well, for instance, if the polling frequency should be 
different in case of tenants.

Best Regards,
Ildiko

 --
 Julien Danjou
 -- Free Software hacker - independent consultant
 -- http://julien.danjou.info
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Julien Danjou
On Wed, Jan 08 2014, Ildikó Váncsa wrote:

 My idea was just about providing the possibility to configure the data
 collection in Ceilometer differently for the different tenants, I didn't
 mean to link it to an API or at least not on the first place. It could be
 done by the operator as well, for instance, if the polling frequency should
 be different in case of tenants.

Yeah, that would work, we would just need to add a list of project to
the yaml file. We are already doing that for resources anyway, we can do
it for user and project as well.

-- 
Julien Danjou
;; Free Software hacker ; independent consultant
;; http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Ildikó Váncsa
Hi,

  My idea was just about providing the possibility to configure the data 
  collection in Ceilometer differently for the different tenants, I 
  didn't mean to link it to an API or at least not on the first place. 
  It could be done by the operator as well, for instance, if the polling 
  frequency should be different in case of tenants.

 Yeah, that would work, we would just need to add a list of project to the 
 yaml file. We are already doing that for resources anyway, we can do it for 
 user and project as well.

Ok, that sounds good. Then I will create a blueprint based on this direction.

Best Regards,
Ildiko

 --
 Julien Danjou
 ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Doug Hellmann
 of that implementation either, so now I’m kind of waiting for
 the outcome of this Dynamic Meters bp.



 As far as the other configuration settings, we had the conversation about
 updating those through some sort of API early on, and decided that there
 are already lots of operational tools out there to manage changes to those
 files. I would need to see a list of which options people would want to
 have changed through an API to comment further.



 ildikov: Yes, I agree that not all the parameters should be configured
 dynamically. It just popped into my mind regarding the dynamic
 configuration, that there would be a need to configure other configuration
 parameters, not just meters, that is why I mentioned it as a considerable
 item.


OK.





 Doug







 Doug






 If it is ok for you, I will register the bp for this per-project tenant
 settings with some details, when I'm finished with the initial design of
 how this feature could work.

 Best Regards,
 Ildiko


 -Original Message-
 From: Neal, Phil [mailto:phil.n...@hp.com]
 Sent: Tuesday, January 07, 2014 11:50 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

 For multi-node deployments, implementing something like inotify would
 allow administrators to push configuration changes out to multiple targets
 using puppet/chef/etc. and have the daemons pick it up without restart.
 Thumbs up to that.

 As Tim Bell suggested, API-based enabling/disabling would allow users to
 update meters via script, but then there's the question of how to work out
 the global vs. per-project tenant settings...right now we collect specified
 meters for all available projects, and the API returns whatever data is
 stored minus filtered values. Maybe I'm missing something in the
 suggestion, but turning off collection for an individual project seems like
 it'd require some deep changes.

 Vijay, I'll repeat dhellmann's request: do you have more detail in another
 doc? :-)

 -   Phil

  -Original Message-
  From: Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
  [mailto:vijayakumar.kodam@nsn.com]
  Sent: Tuesday, January 07, 2014 2:49 AM
  To: OpenStack Development Mailing List (not for usage questions)
  Cc: chmo...@enovance.com
  Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
  From: ext Chmouel Boudjnah [mailto:chmo...@enovance.com]
  Sent: Monday, January 06, 2014 2:19 PM
  To: OpenStack Development Mailing List (not for usage questions)
  Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 
 
 
 
 
  On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata
  Consultancy Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:
 
  In this case, simply changing the meter properties in a configuration
  file should be enough. There should be an inotify signal which shall
  notify ceilometer of the changes in the config file. Then ceilometer
  should automatically update the meters without restarting.
 
 
 
  Why it cannot be something configured by the admin with inotifywait(1)
  command?
 
 
 
  Or this can be an API call for enabling/disabling meters which could
  be more useful without having to change the config files.
 
 
 
  Chmouel.
 
 
 
  I haven't tried inotifywait() in this implementation. I need to check
  if it will be useful for the current implementation.
 
  Yes. API call could be more useful than changing the config files
 manually.
 
 
 
  Thanks,
 
  VijayKumar

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Tim Bell


Thanks for the clarifications. Given the role descriptions as provided, I no 
longer think there is a need for an API call or per project meter 
enable/disable. Thus, the inotify approach would seem to be viable (and much 
simpler to implement since the state is clearly defined across daemon restarts)



Tim


From: Doug Hellmann [mailto:doug.hellm...@dreamhost.com]
Sent: 08 January 2014 19:27
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer



On Wed, Jan 8, 2014 at 12:35 PM, Ildikó Váncsa 
ildiko.van...@ericsson.commailto:ildiko.van...@ericsson.com wrote:
Hi Doug,

Answers inline again.

Best Regards,
Ildiko

On Wed, Jan 8, 2014 at 3:16 AM, Ildikó Váncsa 
ildiko.van...@ericsson.commailto:ildiko.van...@ericsson.com wrote:
Hi,

I've started to work on the idea of supporting a kind of tenant/project based 
configuration for Ceilometer. Unfortunately I haven't reached the point of 
having a blueprint that could be registered until now. I do not have a deep 
knowledge about the collector and compute agent services, but this feature 
would require some deep changes for sure. Currently there are pipelines for 
data collection and transformation, where the counters can be specified, about 
which data should be collected and also the time interval for data collection 
and so on. These pipelines can be configured now globally in the pipeline.yaml 
file, which is stored right next to the Ceilometer configuration files.

Yes, the data collection was designed to be configured and controlled by the 
deployer, not the tenant. What benefits do we gain by giving that control to 
the tenant?

ildikov: Sorry, my explanation was not clear. I meant there the configuration 
of data collection for projects, what was mentioned by Tim Bell in a previous 
email. This would mean that the project administrator is able to create a data 
collection configuration for his/her own project, which will not affect the 
other project's configuration. The tenant would be able to specify meters 
(enabled/disable based on which ones are needed) for the given project also 
with project specific time intervals, etc.

OK, I think some of the confusion is terminology. Who is a project 
administrator? Is that someone with access to change ceilometer's 
configuration file directly? Someone with a particular role using the API? Or 
something else?

ildikov: As project administrator I meant a user with particular role, a user 
assigned to a tenant.

OK, so like I said, we did not design the system with the idea that a user of 
the cloud (rather than the deployer of the cloud) would have any control over 
what data was collected. They can ask questions about only some of the data, 
but they can't tell ceilometer what to collect.

There's a certain amount of danger in giving the cloud user (no matter their 
role) an off switch for the data collection. As Julien pointed out, it can 
have a negative effect on billing -- if they tell the cloud not to collect data 
about what instances are created, then the deployer can't bill for those 
instances. Differentiating between the values that always must be collected and 
the ones the user can control makes providing an API to manage data collection 
more complex.

Is there some underlying use case behind all of this that someone could 
describe in more detail, so we might be able to find an alternative, or explain 
how to use the existing features to achieve the goal? For example, it is 
already possible to change the pipeline config file to control which data is 
collected and stored. If we make the pipeline code in ceilometer watch for 
changes to that file, and rebuild the pipelines when the config is updated, 
would that satisfy the requirements?

In my view, we could keep the dynamic meter configuration bp with considering 
to extend it to dynamic configuration of Ceilometer, not just the meters and we 
could have a separate bp for the project based configuration of meters.

Ceilometer uses oslo.config, just like all of the rest of OpenStack. How are 
the needs for dynamic configuration updates in ceilometer different from the 
other services?

ildikov: There are some parameters in the configuration file of Ceilometer, 
like log options and notification types, which would be good to be able to 
configure them dynamically. I just wanted to reflect to that need. As I see, 
there are two options here. The first one is to identify the group of the 
dynamically modifiable parameters and move them to the API level. The other 
option could be to make some modifications in oslo.config too, so other 
services also could use the benefits of dynamic configuration. For example the 
log settings could be a good candidate, as for example the change of log 
levels, without service restart, in case debugging the system can be a useful 
feature for all of the OpenStack services.

I misspoke earlier. If we're talking about meters

Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
 
 
 From: ext Doug Hellmann [doug.hellm...@dreamhost.com]
 Sent: Wednesday, January 08, 2014 8:26 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 
 
 On Wed, Jan 8, 2014 at 12:35 PM, Ildikó Váncsa 
 ildiko.van...@ericsson.commailto:ildiko.van...@ericsson.com wrote:
 
 Hi Doug,
 
 Answers inline again.
 
 Best Regards,
 
 Ildiko
 
 
 On Wed, Jan 8, 2014 at 3:16 AM, Ildikó Váncsa 
 ildiko.van...@ericsson.commailto:ildiko.van...@ericsson.com wrote:
 
 Hi,
 
 I've started to work on the idea of supporting a kind of tenant/project
  based configuration for Ceilometer. Unfortunately I haven't reached
  the point of having a blueprint that could be registered until now.
  I do not have a deep knowledge about the collector and compute agent
  services, but this feature would require some deep changes for sure.
  Currently there are pipelines for data collection and transformation,
  where the counters can be specified, about which data should be
  collected and also the time interval for data collection and so on.
  These pipelines can be configured now globally in the pipeline.yaml file,
  which is stored right next to the Ceilometer configuration files.
 
 Yes, the data collection was designed to be configured and controlled by
  the deployer, not the tenant. What benefits do we gain by giving that
  control to the tenant?
 
 ildikov: Sorry, my explanation was not clear. I meant there the configuration
  of data collection for projects, what was mentioned by Tim Bell in a
  previous email. This would mean that the project administrator is able to
  create a data collection configuration for his/her own project, which will
  not affect the other project’s configuration. The tenant would be able to
  specify meters (enabled/disable based on which ones are needed) for the given
  project also with project specific time intervals, etc.
 
 OK, I think some of the confusion is terminology.
 Who is a project administrator? Is that someone with access to change
  ceilometer's configuration file directly? Someone with a particular role
  using the API? Or something else?
 
 ildikov: As project administrator I meant a user with particular role,
  a user assigned to a tenant.
 
 
 OK, so like I said, we did not design the system with the idea that a
  user of the cloud (rather than the deployer of the cloud) would have
  any control over what data was collected. They can ask questions about
  only some of the data, but they can't tell ceilometer what to collect.
 There's a certain amount of danger in giving the cloud user
  (no matter their role) an off switch for the data collection.
 
  As Julien pointed out, it can have a negative effect on billing
  -- if they tell the cloud not to collect data about what instances
  are created, then the deployer can't bill for those instances.
  Differentiating between the values that always must be collected and
  the ones the user can control makes providing an API to manage data
  collection more complex.
 
 Is there some underlying use case behind all of this that someone could
  describe in more detail, so we might be able to find an alternative, or
  explain how to use the existing features to achieve the goal?
 
  For example, it is already possible to change the pipeline config file
  to control which data is collected and stored.
  If we make the pipeline code in ceilometer watch for changes to that file,
  and rebuild the pipelines when the config is updated,
  would that satisfy the requirements?
 

Yes. That's exactly the requirement for our blueprint. To avoid ceilometer 
restart for changes to take effect, when the config file changes.
API support was added later based on the request in this mail chain. We 
actually don't need APIs and can be removed.

So as you mentioned above, whenever the config file is changed, ceilometer 
should update the meters accordingly.



 
 
 In my view, we could keep the dynamic meter configuration bp with considering
  to extend it to dynamic configuration of Ceilometer, not just the meters and
  we could have a separate bp for the project based configuration of meters.
 Ceilometer uses oslo.config, just like all of the rest of OpenStack. How are
  the needs for dynamic configuration updates in ceilometer different from
  the other services?
 
 
 ildikov: There are some parameters in the configuration file of Ceilometer,
  like log options and notification types, which would be good to be able to
  configure them dynamically. I just wanted to reflect to that need. As I see,
  there are two options here. The first one is to identify the group of the
  dynamically modifiable parameters and move them to the API level. The other
  option could be to make some modifications in oslo.config too, so other
  services also could use the benefits of dynamic configuration

Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Doug Hellmann
On Wed, Jan 8, 2014 at 2:08 PM, Tim Bell tim.b...@cern.ch wrote:



 Thanks for the clarifications. Given the role descriptions as provided, I
 no longer think there is a need for an API call or per project meter
 enable/disable. Thus, the inotify approach would seem to be viable (and
 much simpler to implement since the state is clearly defined across daemon
 restarts)

Good, thanks, Tim.

Doug






 Tim





 *From:* Doug Hellmann [mailto:doug.hellm...@dreamhost.com]
 *Sent:* 08 January 2014 19:27

 *To:* OpenStack Development Mailing List (not for usage questions)
 *Subject:* Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer







 On Wed, Jan 8, 2014 at 12:35 PM, Ildikó Váncsa ildiko.van...@ericsson.com
 wrote:

  Hi Doug,



 Answers inline again.



 Best Regards,

 Ildiko



 On Wed, Jan 8, 2014 at 3:16 AM, Ildikó Váncsa ildiko.van...@ericsson.com
 wrote:

 Hi,

 I've started to work on the idea of supporting a kind of tenant/project
 based configuration for Ceilometer. Unfortunately I haven't reached the
 point of having a blueprint that could be registered until now. I do not
 have a deep knowledge about the collector and compute agent services, but
 this feature would require some deep changes for sure. Currently there are
 pipelines for data collection and transformation, where the counters can be
 specified, about which data should be collected and also the time interval
 for data collection and so on. These pipelines can be configured now
 globally in the pipeline.yaml file, which is stored right next to the
 Ceilometer configuration files.



 Yes, the data collection was designed to be configured and controlled by
 the deployer, not the tenant. What benefits do we gain by giving that
 control to the tenant?



 ildikov: Sorry, my explanation was not clear. I meant there the
 configuration of data collection for projects, what was mentioned by Tim
 Bell in a previous email. This would mean that the project administrator is
 able to create a data collection configuration for his/her own project,
 which will not affect the other project’s configuration. The tenant would
 be able to specify meters (enabled/disable based on which ones are needed)
 for the given project also with project specific time intervals, etc.



 OK, I think some of the confusion is terminology. Who is a project
 administrator? Is that someone with access to change ceilometer's
 configuration file directly? Someone with a particular role using the API?
 Or something else?



 ildikov: As project administrator I meant a user with particular role, a
 user assigned to a tenant.



 OK, so like I said, we did not design the system with the idea that a user
 of the cloud (rather than the deployer of the cloud) would have any control
 over what data was collected. They can ask questions about only some of the
 data, but they can't tell ceilometer what to collect.



 There's a certain amount of danger in giving the cloud user (no matter
 their role) an off switch for the data collection. As Julien pointed out,
 it can have a negative effect on billing -- if they tell the cloud not to
 collect data about what instances are created, then the deployer can't bill
 for those instances. Differentiating between the values that always must be
 collected and the ones the user can control makes providing an API to
 manage data collection more complex.



 Is there some underlying use case behind all of this that someone could
 describe in more detail, so we might be able to find an alternative, or
 explain how to use the existing features to achieve the goal? For example,
 it is already possible to change the pipeline config file to control which
 data is collected and stored. If we make the pipeline code in ceilometer
 watch for changes to that file, and rebuild the pipelines when the config
 is updated, would that satisfy the requirements?



  In my view, we could keep the dynamic meter configuration bp with
 considering to extend it to dynamic configuration of Ceilometer, not just
 the meters and we could have a separate bp for the project based
 configuration of meters.



 Ceilometer uses oslo.config, just like all of the rest of OpenStack. How
 are the needs for dynamic configuration updates in ceilometer different
 from the other services?



 ildikov: There are some parameters in the configuration file of
 Ceilometer, like log options and notification types, which would be good to
 be able to configure them dynamically. I just wanted to reflect to that
 need. As I see, there are two options here. The first one is to identify
 the group of the dynamically modifiable parameters and move them to the API
 level. The other option could be to make some modifications in oslo.config
 too, so other services also could use the benefits of dynamic
 configuration. For example the log settings could be a good candidate, as
 for example the change of log levels, without service restart, in case
 debugging the system can

Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Doug Hellmann
On Wed, Jan 8, 2014 at 3:09 PM, Kodam, Vijayakumar (EXT-Tata Consultancy
Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:



  
 
  From: ext Doug Hellmann [doug.hellm...@dreamhost.com]
  Sent: Wednesday, January 08, 2014 8:26 PM

  To: OpenStack Development Mailing List (not for usage questions)
  Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
  
  
  On Wed, Jan 8, 2014 at 12:35 PM, Ildikó Váncsa 
 ildiko.van...@ericsson.com wrote:
  
  Hi Doug,
  
  Answers inline again.
  
  Best Regards,
  
  Ildiko
  
  
  On Wed, Jan 8, 2014 at 3:16 AM, Ildikó Váncsa 
 ildiko.van...@ericsson.com wrote:
  
  Hi,
  
  I've started to work on the idea of supporting a kind of tenant/project
   based configuration for Ceilometer. Unfortunately I haven't reached
   the point of having a blueprint that could be registered until now.
   I do not have a deep knowledge about the collector and compute agent
   services, but this feature would require some deep changes for sure.
   Currently there are pipelines for data collection and transformation,
   where the counters can be specified, about which data should be
   collected and also the time interval for data collection and so on.
   These pipelines can be configured now globally in the pipeline.yaml
 file,
   which is stored right next to the Ceilometer configuration files.
  
  Yes, the data collection was designed to be configured and controlled by
   the deployer, not the tenant. What benefits do we gain by giving that
   control to the tenant?
  
  ildikov: Sorry, my explanation was not clear. I meant there the
 configuration
   of data collection for projects, what was mentioned by Tim Bell in a
   previous email. This would mean that the project administrator is able
 to
   create a data collection configuration for his/her own project, which
 will
   not affect the other project’s configuration. The tenant would be able
 to
   specify meters (enabled/disable based on which ones are needed) for the
 given
   project also with project specific time intervals, etc.
  
  OK, I think some of the confusion is terminology.
  Who is a project administrator? Is that someone with access to change
   ceilometer's configuration file directly? Someone with a particular role
   using the API? Or something else?
  
  ildikov: As project administrator I meant a user with particular role,
   a user assigned to a tenant.
  
  
  OK, so like I said, we did not design the system with the idea that a
   user of the cloud (rather than the deployer of the cloud) would have
   any control over what data was collected. They can ask questions about
   only some of the data, but they can't tell ceilometer what to collect.
  There's a certain amount of danger in giving the cloud user
   (no matter their role) an off switch for the data collection.
  
   As Julien pointed out, it can have a negative effect on billing
   -- if they tell the cloud not to collect data about what instances
   are created, then the deployer can't bill for those instances.
   Differentiating between the values that always must be collected and
   the ones the user can control makes providing an API to manage data
   collection more complex.
  
  Is there some underlying use case behind all of this that someone could
   describe in more detail, so we might be able to find an alternative, or
   explain how to use the existing features to achieve the goal?
  
   For example, it is already possible to change the pipeline config file
   to control which data is collected and stored.
   If we make the pipeline code in ceilometer watch for changes to that
 file,
   and rebuild the pipelines when the config is updated,
   would that satisfy the requirements?
  

  Yes. That's exactly the requirement for our blueprint. To avoid
 ceilometer restart for changes to take effect, when the config file
 changes.
 API support was added later based on the request in this mail chain. We
 actually don't need APIs and can be removed.

 So as you mentioned above, whenever the config file is changed, ceilometer
 should update the meters accordingly.


OK, I think that's something reasonable to implement, although I would
have to look at the collector to make sure we could rebuild the pipelines
safely without losing any data as more messages come in. But it should be
possible, if not easy. :-)

The blueprint should be updated to reflect this approach.

Doug





  
  
  In my view, we could keep the dynamic meter configuration bp with
 considering
   to extend it to dynamic configuration of Ceilometer, not just the
 meters and
   we could have a separate bp for the project based configuration of
 meters.
  Ceilometer uses oslo.config, just like all of the rest of OpenStack. How
 are
   the needs for dynamic configuration updates in ceilometer different from
   the other services?
  
  
  ildikov: There are some parameters

Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-08 Thread Ildikó Váncsa
Hi Doug,
OK, so like I said, we did not design the system with the idea that a user of 
the cloud (rather than the deployer of the cloud) would have any control over 
what data was collected. They can ask questions about only some of the data, 
but they can't tell ceilometer what to collect.

There's a certain amount of danger in giving the cloud user (no matter their 
role) an off switch for the data collection. As Julien pointed out, it can 
have a negative effect on billing -- if they tell the cloud not to collect data 
about what instances are created, then the deployer can't bill for those 
instances. Differentiating between the values that always must be collected and 
the ones the user can control makes providing an API to manage data collection 
more complex.

Is there some underlying use case behind all of this that someone could 
describe in more detail, so we might be able to find an alternative, or explain 
how to use the existing features to achieve the goal? For example, it is 
already possible to change the pipeline config file to control which data is 
collected and stored. If we make the pipeline code in ceilometer watch for 
changes to that file, and rebuild the pipelines when the config is updated, 
would that satisfy the requirements?

ildikov: Thanks for the clarification. The base idea was to provide the 
possibility of different data collection configuration for projects. Reflecting 
to the dynamic meter configuration and the possible API based solution, it 
seemed to be possible to provide the configuration possibility to the users of 
the cloud as well. At this point, I haven't considered the billing aspect, 
which would be affected by this extra option, as you mentioned it above, so it 
was definitely a wrong direction. Finally we've reached a consensus with Julien 
by making the required changes in pipeline.yaml and the related codebase.

ildikov: Thanks to both of you for the effort in clarifying this.

Best Regards,
Ildiko
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-07 Thread Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
From: ext Chmouel Boudjnah [mailto:chmo...@enovance.com]
Sent: Monday, January 06, 2014 2:19 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer


On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata Consultancy Ser - 
FI/Espoo) vijayakumar.kodam@nsn.commailto:vijayakumar.kodam@nsn.com 
wrote:
In this case, simply changing the meter properties in a configuration file 
should be enough. There should be an inotify signal which shall notify 
ceilometer of the changes in the config file. Then ceilometer should 
automatically update the meters without restarting.

Why it cannot be something configured by the admin with inotifywait(1) command?

Or this can be an API call for enabling/disabling meters which could be more 
useful without having to change the config files.

Chmouel.

I haven't tried inotifywait() in this implementation. I need to check if it 
will be useful for the current implementation.
Yes. API call could be more useful than changing the config files manually.

Thanks,
VijayKumar
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-07 Thread Neal, Phil
For multi-node deployments, implementing something like inotify would allow 
administrators to push configuration changes out to multiple targets using 
puppet/chef/etc. and have the daemons pick it up without restart. Thumbs up to 
that.

As Tim Bell suggested, API-based enabling/disabling would allow users to update 
meters via script, but then there's the question of how to work out the global 
vs. per-project tenant settings...right now we collect specified meters for all 
available projects, and the API returns whatever data is stored minus filtered 
values. Maybe I'm missing something in the suggestion, but turning off 
collection for an individual project seems like it'd require some deep changes.

Vijay, I'll repeat dhellmann's request: do you have more detail in another doc? 
:-)

-   Phil

 -Original Message-
 From: Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
 [mailto:vijayakumar.kodam@nsn.com]
 Sent: Tuesday, January 07, 2014 2:49 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Cc: chmo...@enovance.com
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 From: ext Chmouel Boudjnah [mailto:chmo...@enovance.com]
 Sent: Monday, January 06, 2014 2:19 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer
 
 
 
 
 
 On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata Consultancy
 Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:
 
 In this case, simply changing the meter properties in a configuration file
 should be enough. There should be an inotify signal which shall notify
 ceilometer of the changes in the config file. Then ceilometer should
 automatically update the meters without restarting.
 
 
 
 Why it cannot be something configured by the admin with inotifywait(1)
 command?
 
 
 
 Or this can be an API call for enabling/disabling meters which could be more
 useful without having to change the config files.
 
 
 
 Chmouel.
 
 
 
 I haven't tried inotifywait() in this implementation. I need to check if it 
 will be
 useful for the current implementation.
 
 Yes. API call could be more useful than changing the config files manually.
 
 
 
 Thanks,
 
 VijayKumar

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-06 Thread Julien Danjou
On Tue, Dec 31 2013, Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo) 
wrote:

Hi,

 Currently there is no way to enable or disable meters without restarting 
 ceilometer.

 There are cases where operators do not want to run all the meters 
 continuously.
 In these cases, there should be a way to disable or enable them dynamically.

 We are working on this feature right now. I have also created a blueprint for 
 the same.
 https://blueprints.launchpad.net/ceilometer/+spec/dynamic-meters

 We would love to hear your views on this feature.

This looks like a good idea, but it was my understanding that it was
pretty much already doable by modifying the configuration file and
reloading the daemons. What else would you have in mind?

-- 
Julien Danjou
/* Free Software hacker * independent consultant
   http://julien.danjou.info */


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-06 Thread Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo)
-Original Message-
From: ext Julien Danjou [mailto:jul...@danjou.info] 
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

On Tue, Dec 31 2013, Kodam, Vijayakumar (EXT-Tata Consultancy Ser - FI/Espoo) 
wrote: 

Hi,

 Currently there is no way to enable or disable meters without restarting 
 ceilometer.

 There are cases where operators do not want to run all the meters 
 continuously.
 In these cases, there should be a way to disable or enable them dynamically.

 We are working on this feature right now. I have also created a blueprint 
 for the same.
 https://blueprints.launchpad.net/ceilometer/+spec/dynamic-meters

 We would love to hear your views on this feature.

This looks like a good idea, but it was my understanding that it was
pretty much already doable by modifying the configuration file and
reloading the daemons. What else would you have in mind?

Changes to the meters or configuration files should reflect immediately without 
restarting daemons.
One scenario where this feature could be useful is, when Ceilometer is 
processing several hundreds of meters and operator decides to disable some 
specific meters without affecting other meters.
In this case, simply changing the meter properties in a configuration file 
should be enough. There should be an inotify signal which shall notify 
ceilometer of the changes in the config file. Then ceilometer should 
automatically update the meters without restarting.

That is the main idea of this blueprint. 

Regards,
VijayKumar

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-06 Thread Chmouel Boudjnah
On Mon, Jan 6, 2014 at 12:52 PM, Kodam, Vijayakumar (EXT-Tata Consultancy
Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:

 In this case, simply changing the meter properties in a configuration file
 should be enough. There should be an inotify signal which shall notify
 ceilometer of the changes in the config file. Then ceilometer should
 automatically update the meters without restarting.


Why it cannot be something configured by the admin with inotifywait(1)
command?

Or this can be an API call for enabling/disabling meters which could be
more useful without having to change the config files.

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-06 Thread Doug Hellmann
On Tue, Dec 31, 2013 at 4:53 AM, Kodam, Vijayakumar (EXT-Tata Consultancy
Ser - FI/Espoo) vijayakumar.kodam@nsn.com wrote:

  Hi,

 Currently there is no way to enable or disable meters without restarting
 ceilometer.

 There are cases where operators do not want to run all the meters
 continuously.
 In these cases, there should be a way to disable or enable them
 dynamically.

 We are working on this feature right now. I have also created a blueprint
 for the same.
 https://blueprints.launchpad.net/ceilometer/+spec/dynamic-meters

 We would love to hear your views on this feature.


There isn't much detail in the blueprint. Do you have a more comprehensive
document you can link to that talks about how you intend for it to work?

Doug




 Regards,
 VijayKumar Kodam




 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer

2014-01-06 Thread Tim Bell

Thinking using inotify/configuration file changes to implement dynamic meters, 
this would be limited to administrators of ceilometer itself (i.e. with write 
access to the file) rather than the project administrators (as defined by 
keystone roles). Thus, as a project administrator who is not the cloud admin, I 
could not enable/disable the meter for a project only.

It would mean that scripting meter on/off would not be possible if there was 
not an API to perform this.

Not sure if these requirements are significant and the associated impact on 
implementation complexity, but they may be relevant in scoping out the 
blueprint and subsequent changes

Tim

From: Doug Hellmann [mailto:doug.hellm...@dreamhost.com]
Sent: 06 January 2014 23:35
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ceilometer] Dynamic Meters in Ceilometer



On Tue, Dec 31, 2013 at 4:53 AM, Kodam, Vijayakumar (EXT-Tata Consultancy Ser - 
FI/Espoo) vijayakumar.kodam@nsn.commailto:vijayakumar.kodam@nsn.com 
wrote:
Hi,

Currently there is no way to enable or disable meters without restarting 
ceilometer.

There are cases where operators do not want to run all the meters continuously.
In these cases, there should be a way to disable or enable them dynamically.

We are working on this feature right now. I have also created a blueprint for 
the same.
https://blueprints.launchpad.net/ceilometer/+spec/dynamic-meters

We would love to hear your views on this feature.

There isn't much detail in the blueprint. Do you have a more comprehensive 
document you can link to that talks about how you intend for it to work?

Doug



Regards,
VijayKumar Kodam




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev