from:"gord chung"

Re: [openstack-dev] [telemetry][ceilometer] New project: collectd-ceilometer-plugin

2016-01-22 Thread gord chung


nice! thanks Emma!

i'm wondering if there's an additional metrics/resources we should add 
to gnocchi to accommodate the data?


On 22/01/2016 6:11 AM, Foley, Emma L wrote:

Hi folks,

A new plug-in for making collectd[1] stats available to Ceilometer [2] is ready 
for use.

The collectd-ceilometer-plugin make system statistics from collectd available 
to Ceilometer.
These additional statistics make it easier to detect faults and identify 
performance bottlenecks (among other uses).

Regards,
Emma

[1] https://collectd.org/
[2] http://github.com/openstack/collectd-ceilometer-plugin
  
--

Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Tip: jsonformatter site for parsing/debugging logs

2016-01-21 Thread gord chung




On 21/01/2016 11:27 AM, Doug Hellmann wrote:

You can also do this using Python's json module from the command line:

$ echo '{"json":"obj"}' | python -m json.tool
{
   "json": "obj"
}

Doug

very useful... if you want a site,  i use http://pro.jsonlint.com/ 
(vladikr told me about it... he will be happy i mentioned his name here.)


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] ceilometer Floatingip Pollster change

2016-01-20 Thread gord chung

i guess i should probably add that the current pollster doesn't work 
because of limitation in nova api[1].
if we want to support discovery of whether nova-network or neutron is 
used, the existing pollster and nova-api still needs to be fixed.


[1] https://bugs.launchpad.net/ceilometer/+bug/1346429

On 19/01/2016 11:28 PM, Pradeep Kilambi wrote:
Ceilometer floatingip pollster currently polls the nova api to get the 
floating ip info periodically. But due to limitations in the nova api 
as listed in this bug[1],
the data ceilometer receives isn't all that useful. Nova doesn't 
return this info for all tenants. There has been some work in juno 
time frame but it was

reverted due to the fact that the nova api logs were being spammed [2].

Due to these concerns, the proposal now is to use the neutron api to 
get this data as proposed in this patch[3]. What we would like to know 
is,



1. Is the data gathered from current floating ip pollster being used? 
if so how and what context ?


2. This might only be an issue for pure nova-network scenario, but 
even that i'm not sure how useful the current data we gather is?


3. Are there any other concerns regarding this change that we should 
discuss?


Any feedback appreciated,


Thanks,
~ Pradeep

[1] https://bugs.launchpad.net/nova/+bug/1402514

[2] 
http://lists.openstack.org/pipermail/openstack-dev/2014-June/037304.html


[3] https://review.openstack.org/#/c/269369/



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] :Regarding wild card configuration in pipeline.yaml

2016-01-20 Thread gord chung


hi,

i don't completely recall why we don't allow wildcarded exclude and 
include meters. it's probably because there's ambiguity of ordering of 
wildcard which can lead to different filter results.


someone can correct me, but i don't think there's a strict requirement 
that stops us from supporting both at once, just that it's not there.


as it stands now. you'll need to explicitly list out the meters you want 
(or don't want) sent to each pipeline.


cheers,


On 20/01/2016 8:39 AM, Raghunath D wrote:

Hi ,
We have one use-case for which we are using ceilometer for getting 
notifications.

We have meter's m1.*,m1.xyz.* and publishers(kafka/udp) as p1 and p2.
i.m1.* notifications/meter data should send to p1 and p2
ii.m1.xyz.* notifications/meter data should send to p1.

We can correlate m1.* as network.* and m1.xyz.* as network.incoming.*
The pipeline.yaml is updated as:
--
sources:
  -name : m1meter
   meters: m1.*,m1.xyz.*
   sinks:
-m1sink

sinks:
 -name: m1sink
  publishers:
   -p1
   -p2


With the above configuration p1 also receives other than m1.xyz.* 
notifications
which are not subscribed by p1.To avoid this 
duplication's,pipeline.yaml is updates as:

-
sources:
  -name : m1meter
   meters: m1.*,!m1.xyz.*
   sinks:
-m1sink
   .
  -name : m2meter
   meters:m1.xyz.*
   sinks:
-m2sink
sinks:
 -name: m1sink
  publishers:
   -p1
   -p2

 -name: m2sink
  publishers:
   -p1
 

This configuration is failing under source rule checking with reason 
"both included and

excluded meters specified.

*Info/Help needed:*
  Do we have any way in ceilometer frame work to achieve this or could 
you provide

us some suggestions how to achieve this in ceilometer.

With Best Regards
RaghunathDudyala
TataConsultancy Services Limited
Mailto: raghunat...@tcs.com
Website: http://www.tcs.com 

Experience certainty. IT Services
Business Solutions
Consulting


=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] ceilometer meter-list output is empty

2016-01-14 Thread gord chung


hi,

please ensure you are running latest code (as you seem to want master)

you are running into 
https://review.openstack.org/#/q/I4765b3b9627983a245aa5521a85ad89e83ab8551


On 14/01/2016 12:10 AM, lichen.hangzhou wrote:

Hi,

I have installed a devstack environment with ceilometer enabled:
enable_plugin ceilometer 
https://git.openstack.org/openstack/ceilometer


I  am trying to run some ceilometer commands to test whether my 
ceilometer is working well and get to know ceilometer as well.
But, no matter how many instances I created and how long I wait, the 
out put for command "ceilometer meter-list" & "ceilometer sample-list" 
are empty.


ceilometer meter-list
+--+--+--+-+-++
| Name | Type | Unit | Resource ID | User ID | Project ID |
+--+--+--+-+-++
+--+--+--+-+-++

By checking devstack log based on my active instance id, I get :

ceilometer-acentral.log:
2016-01-14-170750:2794:2016-01-14 20:58:17.750 20284 
ERROR ceilometer.hardware.discovery 
[req-51230835-1a74-4846-be5e-c032e24ad54f admin - - - -] Couldn't 
obtain IP address of instance d0a781a2-5adc-48c3-8976-f55f6613b68b


  ceilometer-acompute.log:
2016-01-14 18:48:41.030 20885 WARNING 
ceilometer.compute.pollsters.memory 
[req-218d31b1-d0c5-4b7f-99b8-57a8b4a6fc48 admin - - - -] Cannot 
inspect data of MemoryUsagePollster for 
d0a781a2-5adc-48c3-8976-f55f6613b68b, non-fatal reason: Failed to 
inspect memory usage of instance id=d0a781a2-5adc-48c3-8976-f55f6613b68b>, can not get info from libvirt.


2016-01-14 20:58:41.095 20885 DEBUG 
ceilometer.compute.pollsters.disk 
[req-218d31b1-d0c5-4b7f-99b8-57a8b4a6fc48 admin - - - -] 
LibvirtInspector does not provide data for 
PerDeviceReadRequestsRatePollster get_samples 
/opt/stack/ceilometer/ceilometer/compute/pollsters/disk.py:309


Also, an error for service ceilometer-anotification :

2016-01-14 17:54:43.661 2218 DEBUG ceilometer.notification 
[req-93091ecb-c80c-488b-8bc3-aff96a0cb17f - - - - -] Event types from 
trove.instance.exists: trove.instance.exists (ack_on_error=True) 
_configure_main_queue_listeners 
/opt/stack/ceilometer/ceilometer/notification.py:219

Traceback (most recent call last):
  File 
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 
457, in fire_timers

timer()
  File 
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 
58, in __call__

cb(*args, **kw)
  File 
"/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 
214, in main

result = function(*args, **kwargs)
  File "/opt/stack/oslo.service/oslo_service/service.py", line 
660, in run_service

service.start()
  File "/opt/stack/ceilometer/ceilometer/notification.py", 
line 174, in start

self.event_pipe_manager)
  File "/opt/stack/ceilometer/ceilometer/notification.py", 
line 223, in _configure_main_queue_listeners

for new_tar in handler.get_targets(cfg.CONF):
  File 
"/opt/stack/ceilometer/ceilometer/database/notifications.py", line 40, 
in get_targets

for topic in conf.notification_topics]
  File "/opt/stack/oslo.config/oslo_config/cfg.py", line 2009, 
in __getattr__

raise NoSuchOptError(name)
NoSuchOptError: no such option in group DEFAULT: 
notification_topics


But I do not enable trove at all.

Anyone can help me ?




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove][neutron][cinder][swift][ceilometer][nova][keystone][sahara][glance][neutron-lbaas][imm] stylistic changes to code, how do we handle them?

2016-01-12 Thread gord chung

not a fan of these changes since it messes up git blame and it becomes 
harder to track down author if you have questions relating to bug/code.


that said, "if var > 0" is not the same as "if var". in the latter, var 
can be any number (negative or positive) or any other datatype and it'll 
evaluate to True. so... don't blindly do this.


On 12/01/2016 8:51 AM, Amrith Kumar wrote:

I've tagged this message with the projects impacted by a series of change sets:

[trove] https://review.openstack.org/#/c/266220/
[neutron] https://review.openstack.org/#/c/266156/1
[cinder] https://review.openstack.org/#/c/266099/2
[swift] https://review.openstack.org/#/c/266185/1
[ceilometer] https://review.openstack.org/#/c/266211/1
[nova] https://review.openstack.org/#/c/266143/2
[keystone] https://review.openstack.org/#/c/266203/2
[sahara] https://review.openstack.org/#/c/266230/1
[glance] https://review.openstack.org/#/c/266192/1
[neutron-lbaas] https://review.openstack.org/#/c/266181/1

I would like the guidance of the developer community in figuring out how to 
proceed with this change, and changes like this.

The change, in essence changes a construct of the kind:

if var > 0:
... something ...

To

if var:
... something ...

In a couple of cases, it also changes messages from (for example, in Trove)

"Limit value must be > 0" to
"Limit value must be greater than zero"

My question to the ML is this, should stylistic changes of this kind be handled 
in a consistent way across all projects, maybe with a hacking rule and some 
discussion on the ML first? After all, if this change is worthwhile, it is 
worth ensuring that this construct that we are seeking to eliminate, does not 
reenter the code base.

For what it is worth, I agree with Vitaly Grindev [sahara, in review 
https://review.openstack.org/#/c/266230/1]. I think the code before the change 
was more intuitive and readable.

Thanks,

-amrith


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [telemetry][aodh][vitrage] The purpose of notification about alarm updating

2016-01-11 Thread gord chung

so one point i should mention is that the notification we propose on 
removing was originally attended for Ceilometer events. much in same way 
as disabling the aodh-notifier to allow Vitrage to consume alarm 
triggers, it isn't exactly safe for Vitrage to consume the existing 
messages as Ceilometer may also consume it.


the idea with moving it to a discrete notifier (whether zaqar or plain 
oslo.messaging) is so the consumer of message is explicit.


On 10/01/2016 1:30 AM, AFEK, Ifat (Ifat) wrote:

Hi Julien,

I'm a bit confused. This thread started by liusheng asking to remove the 
notifications to the bus[1], and I replied saying that we do want to use this 
capability in Vitrage. What is the notifier plugin that you suggested? Is it 
the same notifier that liusheng referred to, or something else?


[1] https://review.openstack.org/#/c/246727/

Thanks,
Ifat.


-Original Message-
From: Julien Danjou [mailto:jul...@danjou.info]
Sent: Friday, January 08, 2016 11:42 AM

On Thu, Jan 07 2016, AFEK, Ifat (Ifat) wrote:


We have two motivations: one is that we want to get notifications on
every change, but we don't want to register our webhook to each and
every alarm; and the other is that we already listen to the message
bus for other openstack components, so we would like to handle aodh

the same way.

If we disable aodh-notifier, won't it have other impacts on the
system? What if someone else used a webhook, and won't be notified
because we disabled the notifier?

We could have a notifier plugin that would send a notification using
oslo.messaging I guess. That shouldn't be a big deal.

--
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [telemetry] mitaka virtual meetup follow-up

2016-01-08 Thread gord chung


hi folks,

just to recap the meeting item[1], we won't have a meetup this cycle as 
we all have our tasks and there doesn't seem to be any items that are 
controversial. we've agree that going forward, if something does require 
quicker feedback than ML/irc, we'll set up a conference and promote it 
to list before hand.


for those looking for help, whether development or operations, feel free 
to reach out on this list or irc.


have a great weekend.

[1] 
http://eavesdrop.openstack.org/meetings/telemetry/2016/telemetry.2016-01-07-15.00.log.html


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

2016-01-08 Thread gord chung




On 08/01/16 04:49 AM, Julien Danjou wrote:

On Thu, Jan 07 2016, Jay Pipes wrote:


OK, I just watched that. Sorry, still don't see the value that Mimic provides
over unit testing the client interfaces and mocking out the HTTP payloads so
you have strict control over the expectations.

The problem that Glyph noted in the video about the unit test that mocked out
os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I
find it odd that that example was used in discussing the value of Mimic. Bad
mocks are just that: bad mocks. The same false positive failure could easily be
introduced with a typo in the "metadata injection" that Mimic does to inject
failures into the system.

Mimic isn't testing anything on the server side at all so I'm not sure why
folks call it "integration testing". It isn't testing the integration of
anything at all. All it enables is multi-language client library testing, and
see my response to Ben, the surface area it introduces for bugs in the test
platform itself in my opinion outweigh the multi-language value it might have.

I completely agree with what Jay points out here.

To illustrate with what we do in the Telemetry team: in order test
gnocchiclient, we pip install gnocchi¹, configure it briefly and use the
client against that freshly installed server. It works very well, and
we're sure we test against real data. That's what I would call
integration testing.
We could also easily test against older version of the Gnocchi server by
pip install-ing an older version if we wanted.

This has way more value than mocking the whole HTTP server and crossing
fingers we did a good job mimicking it. Oh, and it's also actually less
work. :)

¹  https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh



this happens with aodhclient as well. the one caveat i would mention is 
that it doesn't really allow for test isolation which requires you to 
put a bit more thought into your tests.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [telemetry][aodh][vitrage] The purpose of notification about alarm updating

2016-01-06 Thread gord chung




On 06/01/2016 8:11 AM, AFEK, Ifat (Ifat) wrote:

Hi,

We would like to be notified once an alarm state is changed, so we prefer using 
the message bus.
As far as I understand, ceilometer and aodh APIs do not provide immediate 
notifications. If we use the APIs, we will have to poll the data periodically 
and look for changes, and we will get them in delay. By listening to the 
message bus we can get the notifications immediately, like we do with other 
openstack components.

Is there an advantage in using the API instead?
And what do you mean by "aggregation on events data"? what kind of aggregations 
can we do?

Thanks,
Ifat.


is the idea that you don't want to use a webhook to be notified? when an 
alarm is computed by aodh-evaluator, it sends this alarm (via message 
queue) to aodh-notifier which in turns does the webhook. if you just 
disable aodh-notifier, the alarm won't be consumed and you can listen to 
it yourself.


alternatively, it was discussed that maybe adding a zaqar notifier would 
be useful. that way, aodh-notifier would send alarm to zaqar and you 
could configure it to requeue or maybe send a smtp.


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer][Gnocchi] inconsistent instanceattributes cause infinite update

2015-12-21 Thread gord chung

shall we try to synchronise the two datapoints more than? i think that'd 
be the better route since pollster and notification values are 
attempting to meter the same thing.


On 21/12/2015 8:33 AM, Luo Gangyi wrote:

gord, I have looked your link, seems related.
But still, image_ref is a problem.


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [telemetry] cancelling upcoming meetings (24th and 31st)

2015-12-21 Thread gord chung


hi folks,

before i forget, we'll be skipping meetings the next two weeks (24th and 
31st).  we'll resume regularly scheduled meetings on Jan 7th.


please raise anything important to the mailing list (as normal).

have a nice holidays.

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer][Gnocchi] inconsistent instance attributes cause infinite update

2015-12-21 Thread gord chung

we actually looked at this problem here: 
https://review.openstack.org/#/c/235702/


i think we should add support of new attribute instead.

On 20/12/2015 10:52 AM, Luo Gangyi wrote:

Hi devs,
I found a problem which may cause infinite update of instance's 
attributes in gnocchi.

Let's see the resource definition of instance.
  - resource_type: instance
metrics:
  - 'instance'
  - 'memory'
  - 'memory.usage'
  - 'memory.resident'
  - 'vcpus'
  - 'cpu'
  - 'cpu_util'
  - 'disk.root.size'
...
attributes:
  host: resource_metadata.host
  image_ref: resource_metadata.image_ref_url
...
Here is the problem, although they have same  attributes, they are 
*not* same.
Some of them came from nova's notifications and the others are came 
from ceilometer-compute-agent.

1) Those came from notifications, their attributes looks like
image_ref :http://10.133.12.125:9292/images/
host: compute.lgy-openstack-kilo.novalocal
2) Those came from ceilometer-compute-agent,
image_ref : 
http://10.133.12.125:8774/4994e42421a04beda56fff7d817e810e/images/8d6a9cd9-48ae-4a41-bd13-262a46c93d72 


host:ea8f8e465d9caff06e80a0fda6f30d02725e0b55dc0fd940954cb55c
Such difference will cause alternately and infinitely update of a 
instance's attributes if we enable nova audit.
So I suggest we seperate these meters which came from notifications to 
another resource type like "instance_from_notification".

Any other idea?

--
Luo Gangyi
luogan...@chinamobile.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer][gnocchi][vitrage] 'bad' resource_id

2015-12-17 Thread gord chung




On 17/12/15 08:17 AM, AFEK, Ifat (Ifat) wrote:

What do you mean by "create new types of resources"?
Suppose we want to add a new resource type with a new id format, what
should we do for that? Change Gnocchi code / add a plug-in / add a
new type definition in the database / ...?


i believe this comment was because we currently don't have a way to 
support dynamic creation of new resource types. if you want to just add 
a new resource sans metadata, you probably don't need a new type, but if 
you want to capture specific metadata attributes, you will probably need 
to create a new resource in the db.


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer][gnocchi] 'bad' resource_id

2015-12-16 Thread gord chung

On 16/12/2015 4:24 PM, Lu, Lianhao wrote:

On Dec 16, 2015 14:13, Chris Dent wrote:

On Wed, 16 Dec 2015, Lu, Lianhao wrote:

In ceilometer, some metrics(e.g. network.incoming.bytes for VM net
interface, hardware.network.incoming.bytes for host net interface,
compute.node.cpu.percentage for nova compute node host cpu
utilization,
etc.) don't have their resource_id in UUID format(which is required
by gnocchi). Instead, they have something like . as
their resource_id, in some cases even the part won't be in uuid
format. Gnocchi will treat these kind of resource_id as bad id, and
build a new UUID format resource_id for them. Since users are mostly
using resource_id to identify their resources, changing user passed
in resource_id would require the users extra effort to identify the
resources in gnocchi and link them with the resources they original
passed in.

Just for the sake of completeness can you describe the use cases where
the resource_id translation that gnocchi does does not help the use
case. The one way translation is used in the body of search queries as
well as in any URL which contains a resource_id.

I'm sure there are use cases where it breaks down, but I've not heard
them enumerated explicitly.

I'm not saying the translation will break down anything. It's just that in the
case of using ceilometer/gnocchi together, when ceilometer samples are stored
into gnocchi, the users need to do extra steps to figure out which resource to
query to find its related metrics in the bad resource_id case. By simply
looking at the
http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html , the
users can not easily identify the resource and its related metrics in gnocchi.
The users need to be able to do a resource search based on resource attributes,
such as original_resource_id, because in ceilometer/gnocchi cases, they don't
get a chance to see the new resource_id gnocchi generated unless they search.

Say we have configured nova to send out compute node metrics notification which will be turns into
compute.node.cpu.percentage samples by ceilometer and stored into gnocchi, the original resource_id would be
constructed as _ of the nova compute node machine.
But when admin want to search that resource in gnocchi, he either search for a specific new type of resource with
some conditions or search for a generic resource with condition of original_resource_id="_", otherwise he doesn't have ways to find the resource which is identified by
the original resource_id.
but when you query, you do use the original resource_id. the
translation happens on both writes and reads. while in reality, the db
is will store a different id, users shouldn't really be aware of this.

that said, because of pecan, translations don't help when our ids have
'/' in them... we should definitely fix that.

--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] status of distil?

2015-12-14 Thread gord chung


wasn't aware this project existed. i've contacted flwang.

just for reference, for those that extend Telemetry projects, it's good 
to promote it here: 
https://wiki.openstack.org/wiki/Telemetry#Externally_Managed


On 14/12/2015 4:09 AM, Andreas Jaeger wrote:

On 12/14/2015 10:01 AM, Steve Martinelli wrote:

While I was trying to submit patches for projects that had old
keystoneclient references (distil was one of the projects), I noticed
that there hasn't been much action on this project [0]. It's been a year
since a commit [1], no releases [2], and I can't submit a patch since
the .gitreview file doesn't point to review.openstack.org [3].

Is distil alive?

[0] https://github.com/openstack/distil
[1] https://github.com/openstack/distil/commits/master
[2] https://github.com/openstack/distil/releases
[3] https://github.com/openstack/distil/blob/master/.gitreview



There was not a single commit since the project has been imported over 
a year ago into our CI, I suggest it's time to declare it orphaned and 
retire it...


Andreas


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [telemetry][aodh][vitrage] The purpose of notification about alarm updating

2015-12-11 Thread gord chung


sorry, my email was blocking the list for a bit so i missed this.

the original reason this was implemented i believe was to track alarm 
state changes as an event in ceilometer events. i still see this as a 
valid use case but it does duplicate some of the functionality of alarm 
history.


i think the main items are to not flood the message bus with messages 
that no one is listening to. but if there is a use case for it, i'm 
happy to see it remain (and improved).


you can check the patch that Liusheng referenced, but the basic concept 
is as mentioned: send message when an alarm state is changed.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cleaning Up Deprecation Warnings in the Gate

2015-12-10 Thread gord chung




On 10/12/15 12:04 PM, Matthew Treinish wrote:

Using this query to see how many deprecation warning we emit on jobs running on
master in the past 7 days:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22deprecated%5C%22%20AND%20loglevel:%5C%22WARNING%5C%22%20AND%20build_branch:%5C%22master%5C%22

it found 17576707 hits. I don't really see any reason for us to be running dsvm
jobs on master using deprecated options.

We're looking for some people to volunteer to help with this, it isn't something
that's very difficult to fix.
we're discussing it over at #openstack-keystone. this should stop some 
of them: https://review.openstack.org/#/c/256078/



i think most of them are coming from apache/keystone.txt

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db][sqlalchemy][mistral] Configuring default transaction isolation level

2015-12-08 Thread gord chung

we had this in ceilometer events. you can see it here: 
https://github.com/openstack/ceilometer/commit/898cd3d036c4358aa16f7b3e2028365dc9829213


note, that patch is removing it because it slowed everything way down 
because of locking. if you can avoid it, avoid it.


On 08/12/2015 7:28 AM, Renat Akhmerov wrote:

Hi,

Moshe, thanks a lot for bringing this up. I remember I tried to find a 
way to change isolation level per connection but also was unable to do 
that.


An advice from oslo team would be very helpful.

Renat Akhmerov
@ Mirantis Inc.



On 08 Dec 2015, at 13:41, ELISHA, Moshe (Moshe) 
> wrote:


Hi,
We at Mistral want to move from the default transaction isolation 
level of REPEATABLE READ to READ COMMITTED as part of a bugfix[1].
I did not find a way to pass the isolation level to sqlachemy using 
oslo.db and the current solution is to use monkey-patching[2] that 
adds the “isolation_level” property.

Is there currently a better way to set the default isolation level?
If not – I will create a BP for it.
Thanks.
[1]https://review.openstack.org/#/c/253819
[2]https://review.openstack.org/#/c/253819/11/mistral/db/sqlalchemy/base.py
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:openstack-dev-requ...@lists.openstack.org 
?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [telemetry] upcoming Mitaka-1 release

2015-11-26 Thread gord chung


hi,

just a quick note, i'll be tagging the Mitaka-1 release early next week 
for Ceilometer and Aodh. if you want to get something in for M-1, now is 
the time. please let us know so we can track it.


Gnocchi will not have a release -- it will continue to be released 
independently as features are added.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][stable][keystone] keystonemiddleware release 1.5.3 (kilo)

2015-11-24 Thread gord chung

this is the history Brant provided: 
http://lists.openstack.org/pipermail/openstack-dev/2014-May/036427.html


it's apparently on his todo list... or until he finds the person he 
needs to defer it to[1]


[1] 
http://eavesdrop.openstack.org/irclogs/%23openstack-dev/latest.log.html#t2015-11-24T15:48:42




On 24/11/15 02:37 PM, Morgan Fainberg wrote:
Would it be possible to get a bit more detail in (especially if other 
projects are mocking like this) what is being done so that I [or 
anothe rKeystone dev] can work towards a real mock/test module in 
keystonemiddleware so this doesn't occur again due to 
internal-interface mocking?


On Tue, Nov 24, 2015 at 11:24 AM, gord chung <mailto:g...@live.ca>> wrote:


mriedem and myself resolved this with keystone folks earlier
today[1]. should be all better now [2]

[1]

http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2015-11-24.log.html#t2015-11-24T14:38:39
[2] https://review.openstack.org/#/c/249268/



On 24/11/15 12:33 PM, Alan Pevec wrote:

keystonemiddleware 1.5.3: Middleware for OpenStack Identity

periodic-ceilometer-python27-kilo started failing after this
release
First bad:

http://logs.openstack.org/periodic-stable/periodic-ceilometer-python27-kilo/40c5453/testr_results.html.gz
test_acl_scenarios failing with 401 Unauthorized


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 
gord




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][stable][keystone] keystonemiddleware release 1.5.3 (kilo)

2015-11-24 Thread gord chung

mriedem and myself resolved this with keystone folks earlier today[1]. 
should be all better now [2]


[1] 
http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2015-11-24.log.html#t2015-11-24T14:38:39

[2] https://review.openstack.org/#/c/249268/


On 24/11/15 12:33 PM, Alan Pevec wrote:

keystonemiddleware 1.5.3: Middleware for OpenStack Identity

periodic-ceilometer-python27-kilo started failing after this release
First bad: 
http://logs.openstack.org/periodic-stable/periodic-ceilometer-python27-kilo/40c5453/testr_results.html.gz
test_acl_scenarios failing with 401 Unauthorized

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer][aodh][vitrage] Raising custom alarms in AODH

2015-11-23 Thread gord chung




On 23/11/2015 11:14 AM, AFEK, Ifat (Ifat) wrote:

I guess I would like to do both: create a new alarm definition, then
trigger it (call alarm_actions), and possibly later on set its state
back to OK (call ok_action).
I understood that currently all alarm triggering is internal in AODH,
according to threshold/events/combination alarm rules. Would it be
possible to add a new kind of rule, that will allow triggering the
alarm externally?

what type of rule?

i have https://review.openstack.org/#/c/247211 which would theoretically 
allow you to push an action into queue which would then trigger 
appropriate REST call. not sure if it helps you plug into Aodh easier or 
not?


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer][aodh][vitrage] Raising custom alarms in AODH

2015-11-23 Thread gord chung

hi Ifat,

i added some questions below.

On 23/11/2015 7:16 AM, AFEK, Ifat (Ifat) wrote:

Hi,

We have a couple of questions regarding AODH alarms.

In Vitrage[1] project we have two use cases that involve Ceilometer:

1. Import Ceilometer alarms, as well as alarms and resources from other sources
(Nagios, Zabbix, Nova, Heat, etc.), and produce RCA insights about the
connection between different alarms.
to clarify, Ceilometer alarms is deprecated for Aodh and will be removed
very, very soon.

2. Raise "deduced alarms". For example, in case we detect a high memory consumption on a
host, we would like to raise deduced alarms saying "instance might be suffering due to high
memory consumption on the host" on all related instances. Then, we can further deduce that
applications running on these instances might also be affected, and raise alarms on them as well.

Initially we planned to raise these deduced alarms in AODH, so other Openstack
components may consume them as well. Then, when we looked at AODH alarms
documentation, we noticed that there is currently no way of raising custom
alarms. We saw only three types of alarms: threshold alarms, combination alarms
and event alarms.

So, our questions are:

* Is there an alternative way of raising alarms in AODH?
what do we mean by raising alarms? do you want to create a new alarm
definition for Aodh or do you want to trigger an action? do you want to
have a new non-REST action?

* Do you think custom alarms belong in AODH? Are you interested in adding this
capability to AODH?

We would be happy to hear your vision and thoughts about it.

Thanks,
Ifat and Alexey.

[1] https://wiki.openstack.org/wiki/Vitrage

--
gord

Re: [openstack-dev] [Ceilometer]:Subscribe and Publish Notification frame work in Ceilometer !

2015-11-23 Thread gord chung




On 22/11/2015 10:04 PM, Srikanth Vavilapalli wrote:


We have found a related blueprint *[1],* which is exactly addressing 
this dynamic provisioning of pipelines via an API and making use of a 
new data store to store this configuration. But seems like that 
blueprint was abandoned due its limited usage. We agree that it may 
not be that frequent operation to change the pipeline configuration in 
all deployments, but in certain deployments, where applications desire 
to turn ON/OFF receiving these notifications based on some other 
trigger (like anomaly detection, where application might have sensed 
an anomaly based on some set of meters and wanted to turn ON few 
additional set of event/meters to confirm that and turn OFF once the 
anomaly is confirmed).


you're right, we abandoned that because of lack of usage and the idea 
was that generally you'd probably be some type of admin if you were 
editing pipeline file(s).


also, the scope was a big reason for dropping. i think modification of 
configuration via API is not just a Ceilometer requirement but a global 
OpenStack requirement. that said, it's suggested to support this level 
of functionality, it should probably be done in a way that is useful to 
all OpenStack


this cross-project item might be of interest: 
http://lists.openstack.org/pipermail/openstack-dev/2015-November/078400.html


cheers,

--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

2015-11-23 Thread gord chung

given John's reply, i think we should start a new thread? apologies for 
hijacking log thread.


On 23/11/2015 5:01 AM, Alexis Lee wrote:

gord chung said on Fri, Nov 20, 2015 at 01:32:02PM -0500:

On 20/11/15 11:33 AM, Alexis Lee wrote:

why would a producer spit out non-useful datapoints? If no-one cares
or will ever care, it simply shouldn't be included.

... right now the
producer is just sending out a grab bag of data that it thinks is
important but doesn't define who the audience is. ...

... whoever the producer of notifications is, it should know it's
audience.

Here I, with cdent, have to disagree. The criterion has to be
"potentially useful to someone", rather than tailoring production to the
known audience.


so i'm not suggesting producers need to know the entire audience or for 
it to be tailored to a single consumer. i do think whatever we put out 
as a notification should have at least one potential recipient in mind 
(see: useful data to someone [not a hypothetical]). i'm all for 
flexibility and multiple/any consumers, but right now, to be honest, our 
notification criteria seems to be similar to "that guy outside your 
local mall yelling stuff and hoping he makes a convert."


this is just me trying to view it from another approach -- trying to vet 
our current process :)





The problem is knowing what each consumer thinks is interesting and that
isn't something that can be handled by the producer.

^ this is important.


the notification consumption service in ceilometer is essentially
just a pipeline that normalises select incoming notifications into a
data model(s) and pushes that model to whoever wants it (a known
consumer is the storage service but it's configurable to allow other
consumers).

It sounds like Ceilometer is performing a mapping function then and has
to be responsible for knowing how to do that. IE Ceilometer has to
express an opinion on relevance and presentation. This is inevitably
going to be a pain in the butt to maintain, but it's Ceilometer's job,
not the services, since it's Ceilometer trying to express an opinion.
fair enough, i'm just re-raising common feedback i receive when projects 
want to add metrics for ceilometer to collect: 'why do i need to edit 
ceilometer, it's my data.' so there is obviously a divide in the 
community as to whose responsibility it is for maintaining the data.


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

2015-11-23 Thread gord chung




On 20/11/2015 5:12 PM, Chris Dent wrote:

On Fri, 20 Nov 2015, gord chung wrote:

i think a lot of the complexity we have in versioning is that the 
projects are too silo'd. i think some of the versioning issues would 
be irrelevant if the producer knew it's consumers before sending 
rather than producers just tossing out a chunk of data (versioned 
schema or not) and considering their job complete once it leaves it's 
own walls. the producer doesn't necessarily have to be the individual 
project teams but whoever the producer of notifications is, it should 
know it's audience.


To me this is entirely contrary to the point of having notifications
as a generic concept in the OpenStack environment. To me the point is
to ensure that it is possible for the audience to be and do _anything_.


you can still do anything and add anything to notifications but you also 
know that "attributeA/B/C are probably useful to consumerX; 
attributeD/E/F are related and all the attributes can be used by anyone."




We've become so accustomed to some of the misfeatures in the messaging
architecture that we've lost track of the fact that it could be an
event pool on which we have diverse listeners that the producers have
no requirement to know anything about. We could have nova-compute 
spinning

along shouting "I made a VM. I made another VM. Hey, I made another
VM" and "This VM is hot. Halp, I am oversubscribed." All sorts of
tools and devices need to be able to hear that stuff and choose for
themselves what they might do with it.

(This is similar to the reason we have well-formed HTTP APIs: It is so
we can have unexpected clients that do unexpected things.)


i actually view notifications different from 'well-formed HTTP APIs' as 
the initiator is not the consumer but the producer but i agree with 
"unexpected clients that do unexpected things."




It is certainly the case that if we're going to have schematized
and versioned notifications it is critical that the schema are
discoverable in a language independent fashion.

Sometimes it is hard, though, to be convinced that such formalisms are
quite what we really need. From the consumers end the concern is "do
you have these several keys that I care about?" and, as has been said,
the rest is noise. It sounds like microversioned notifications which
almost never version on the major axis might be able to provide this.

We can't allow the introduction of either the formalisms or
discoverability thereof to grant license to change stuff willy nilly.


agree, following this makes versioning pretty much moot. it's the major 
version changes i'm thinking of. how does producer know not to remove 
attributeX? how does consumer know attributeX is now attributeX.Y.Z if 
the name/location is different?



Nor should we be building formalisms that are defenses against an
architecture that's sub-optimal. We need to evolve the formalisms and
the architecture toward the ideal.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

2015-11-20 Thread gord chung




On 20/11/15 11:33 AM, Alexis Lee wrote:

gord chung said on Thu, Nov 19, 2015 at 11:59:33PM -0500:

just to clarify, the idea doesn't involve tailoring the notification
payload to ceilometer, just that if a producer is producing a
notification it knows contains a useful datapoint, the producer
should tell someone explicitly 'this datapoint exists'.

I know very little about Nova notifications or Ceilometer, so stepping
wildly into the unknown here but... why would a producer spit out
non-useful datapoints? If no-one cares or will ever care, it simply
shouldn't be included.

fully agree.

it seems like even before addressing versioning, that the notification 
paradigm itself should be discussed. right now the producer is just 
sending out a grab bag of data that it thinks is important but doesn't 
define who the audience is. while that makes it extremely flexible so 
that anyone can consume the message, it also guarantees nothing (not 
even that it's being consumed). you can version a payload or make a 
schema accessible as much as you like but if no one is listening or the 
data published isn't useful to those listening, it's just noise.


i think a lot of the complexity we have in versioning is that the 
projects are too silo'd. i think some of the versioning issues would be 
irrelevant if the producer knew it's consumers before sending rather 
than producers just tossing out a chunk of data (versioned schema or 
not) and considering their job complete once it leaves it's own walls. 
the producer doesn't necessarily have to be the individual project teams 
but whoever the producer of notifications is, it should know it's audience.




The problem is knowing what each consumer thinks is interesting and that
isn't something that can be handled by the producer. If Ceilometer is
just a pipeline that has no opinion on what's relevant and what isn't,
that's a special case easily implemented by an identity function.
the notification consumption service in ceilometer is essentially just a 
pipeline that normalises select incoming notifications into a data 
model(s) and pushes that model to whoever wants it (a known consumer is 
the storage service but it's configurable to allow other consumers).


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer]:Subscribe and Publish Notification frame work in Ceilometer !

2015-11-19 Thread gord chung




On 19/11/15 10:26 PM, Srikanth Vavilapalli wrote:


Hi Gord

On your second point, Yes, Ceilometer does provide a framework to 
capture a notification and republish to multiple “publish targets” in 
addition to the collector service using udp/kafka/notification as the 
transport mechanisms… We believe this is how “Event Alarm Evaluator” 
module in Aodh project get notified directly from Notification Agents.


However seems like the configuration of these additional “publish 
targets” is supported only through updating the pipeline_cfg_file and 
restarting the corresponding ceilometer services. i.e. the users need 
to manually update the pipeline config files to insert their “publish 
targets” in the sink-publisher configuration for a set of event 
filters of their interest. This type of provisioning is very static.


As per our understanding, ceilometer currently does not provide means 
for users to dynamically register/unregister their “publish targets” 
with ceilometer framework for a subset events of their interest? i.e 
User invokes a ceilometer API with a set of event filters and 
associated publish targets, that can be stored in a data store, which 
will eventually be used by the ceilometer Publisher to dispatch the 
notification to those configured destinations in addition to the 
statically configured “publish targets”. Plz let us know if our 
understanding is wrong or if there are any other means to achieve the 
above functionality. We believe this as a very key functionality 
needed to build latency sensitive (sub-second) analytics application 
on-top of ceilometer framework. We are seeking the feedback from 
community on having this kind of functionality inside ceilometer 
before proceeding with blueprint submission.


actually, in Liberty you can configure a reload[1][2] to refresh 
pipeline when a change happens. based on survey results, most operators 
don't need this functionality so it's off by default but it seems you do. :)


[1] 
https://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/reload-file-based-pipeline-configuration.html
[2] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/pipeline.py#L50-L58


cheers,

--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

2015-11-19 Thread gord chung

On 19/11/15 08:53 PM, Matt Riedemann wrote:

On 11/19/2015 5:52 PM, gord chung wrote:

ceilometer cares. we listen to all notifications and build Measurement
and Event data from (some of) them. to be honest, i don't know if
changes in nova notifications have/are broken in ceilometer because
quite frankly there are far too many notifications to test and track.

as a consumer of data, we don't really care what the format of the
entire payload is. we just care that the select attributes we do care
about are present and in some known position.

in ceilometer, we have two mapping files which define the notifications
we are interested in and how we to normalise them[1][2]. currently, they
are two files contained within ceilometer which we load at runtime. that
said, at the summit we talked about how the projects should own their
own definitions rather than ceilometer[3]. essentially, producer and
consumer schemas are tightly coupled.

the basic premise was that we as consumers have no idea what is getting
produced, but the projects do know and they know exactly what is useful
in what they are producing. so if nova owned their own meters.yaml and
events.yaml, nova devs could easily edit the appropriate file themselves
when they add/modify/remove notification payloads. if a new metric is
added to a notification, it can easily be added to meters.yaml; if a
metric value is moved, meters.yaml can be updated for old and new
locations.

using each projects definition files, ceilometer would load them all in
and as a consumer, not have to worry about what every single change that
happens outside of its domain. it's a different take on it: instead of
having ownership on consumer to interpret accurately, the ownership is
on producer to ensure consumer can understand.

we are working on a POC currently as there some logistics to be thought
out still(ie. how ceilometer knows where to look for each projects
definition files) but using this idea, we wouldn't even need to know
what version of notification we were getting, just that we are told that
we should look "either here or here...". if there's interest in
exploring this idea or if there are any glaring issues please let us
know.

[1]
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml

[2]
https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml

[3] https://etherpad.openstack.org/p/mitaka-telemetry-cross-project

On 19/11/2015 5:05 PM, Ryan Rossiter wrote:

Reading through [1] I started getting worries in the back of my head
about versioning these notifications. The main concern being how can
the consumer know about the versions and what's different between
them? Because these versioned notification payloads hold live nova
objects, there can be a lot of rug-pulling going on underneath these
notifications. If the payload doesn't pin itself to a certain level of
the object, a consumer can never be guaranteed the version of the
payload's object they will be receiving. I ran through a few of the
scenarios about irregular versions in the notifications subteam
meeting on Tuesday [2].

My question is do we care about the consumer? Or is it a case of
"the consumer is always right" so we need to make sure we hand them
super consistent, well-defined blobs across the wire? Consumers will
have no idea of nova object internals, unless they feel like `python
-c import nova`. I do think we get one piece of help from o.vo though.
When the object is serialized, it hands the version with the object.
So consumers can look at the object and say "oh, I got 1.4 I know what
to do with this". But... they will have to implement their own compat
logic. Everyone will have to implement their own compat logic.

We could expose a new API for getting the schema for a specific
version of a notification, so a consumer will know what they're
getting with their notifications. But I think that made mriedem
nauseous. We could make an oslo library that stuffs a shim in between
o.vo and nova's notifications to help out with compat/versioning, but
that sounds like a lot of work, especially because the end goal is
still not clearly defined.

Thoughts?

[1] https://review.openstack.org/#/c/247024
[2]
http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29

I don't think nova is going to maintain it's own meters/events yaml
files in the form that ceilometer necessarily expects/uses today,
since there are other things out there that consume nova notifications
besides ceilometer. But I think you're thinking along the same lines
of what Ryan was thinking, i.e. somehow nova providing a schema or
some way for the consumer to digest/interpret what they are getting,
but I don't know what that would look like or how it would be used by
the consumer.

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

2015-11-19 Thread gord chung

as a consumer of data, we don't really care what the format of the
entire payload is. we just care that the select attributes we do care
about are present and in some known position.

[1]
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml
[2]
https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml

[3] https://etherpad.openstack.org/p/mitaka-telemetry-cross-project

On 19/11/2015 5:05 PM, Ryan Rossiter wrote:
Reading through [1] I started getting worries in the back of my head
about versioning these notifications. The main concern being how can
the consumer know about the versions and what's different between
them? Because these versioned notification payloads hold live nova
objects, there can be a lot of rug-pulling going on underneath these
notifications. If the payload doesn't pin itself to a certain level of
the object, a consumer can never be guaranteed the version of the
payload's object they will be receiving. I ran through a few of the
scenarios about irregular versions in the notifications subteam
meeting on Tuesday [2].

Thoughts?

[1] https://review.openstack.org/#/c/247024
[2]
http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29

--
gord

Re: [openstack-dev] [aodh] HBase gate job

2015-11-17 Thread gord chung


let's change this gate to experimental for now.

On 17/11/15 05:10 AM, Julien Danjou wrote:

Hi,

So we recently decided to run the functional tests for the HBase driver
and enabled the gate job. It turned out the gate job didn't worked, so I
tried to fix it. Now it's almost fixed, and it run the functional tests
and Gabbi tests against Aodh with HBase as a back-end:

   https://review.openstack.org/#/c/245585/

Obviously, as you know this is not a real HBase back-end, but a
in-memory backend that is provided inside Aodh. Now, it turns out that
this driver or this in-memory backend has a bug, as the Gabbi tests
fail.

I don't have time nor interest to fix that, so the way I see it it's
either:
1. Someone stands up, determines if the issue is in the driver or the
in-memory implementation and fix it
2. Nobody cares and we drop HBase gate and deprecates the driver

If nobody stands up in the next week, I'll start working on 2.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [oslo] Please stop removing usedevelop from tox.ini (at least for now)

2015-11-16 Thread gord chung




On 16/11/2015 11:13 AM, Doug Hellmann wrote:

  does anyone want to sign up to remove that last
namespace package?

already in action: https://review.openstack.org/#/c/243255/

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic] Do we need to have a mid-cycle?

2015-11-16 Thread gord chung

in Liberty, we did a virtual meetup in for the telemetry team so i 
thought i'd share experience.


On 16/11/2015 9:05 AM, Jim Rollenhagen wrote:

Then we can set up some hangouts (or similar) to get people in the same
"room" working on things. Time zones will get weird, but we tend to
hangouts is limited to 10 people -- you will hit the capacity. arguably 
not everyone needs to be on video or will talk but you should probably 
find a service that accommodates your expected attendance.



split into smaller groups at the midcycle anyway; this is just more
timezone-aligned. We can also find windows where time zones overlap when
we want to go across those boundaries. Disclaimer: people may need to
work some weird hours to do this well.
ask your company for a coffee budget, a few of you are going to be 
working at 3am.


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][all] Oslo libraries dropping python 2.6 compatability

2015-11-16 Thread gord chung

do we require a major versioning bump for this? i'm wondering how 
'breaking' this is in real life if all the services have dropped py2.6 
already. maybe this just requires an upper-cap in appropriate stable 
branches?


to be devils advocate, one (arguably terrible) reason for not doing a 
major bump is that a lot of libs just did one for the Mitaka cycle.


On 11/11/2015 2:14 PM, Davanum Srinivas wrote:

Folks,

Any concerns? please chime in:
https://review.openstack.org/244275

Thanks
-- Dims



--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [telemetry][aodh][ceilometer][gnocchi] the story of telemetry

2015-11-13 Thread gord chung


hi,

you may have noticed that changes are in motion and an extra telemetry 
tag being floated around. the following is your coles|cliff|spark notes 
on what you need to know.



-- background --
three years ago, the ceilometer project was created to tackle the 
billing of cloud environments. over time, people realised 'data is data 
is data' and anything can be derived from it and ceilometer's scope 
creeped. unfortunately, the scope creep in the developer space could not 
keep up with the scope creep in the user space and caused confusion. it 
also made it difficult to pinpoint issues as the 'ceilometer' term now 
covered so much space.



-- what happened --
in the past few cycles, the ceilometer project has been componentised 
into smaller, discrete projects to tackle specific functionality within 
the telemetry use case domain.



-- where we are now --
to avoid the overloaded term of ceilometer, the Telemetry project[1] was 
created to encapsulate the collective of resulting projects contributing 
to portions of this domain. to clarify the purpose of each individually 
managed project, the following are projects under the Telemetry managed 
domain:


- Aodh - an alarming service[2]
- Ceilometer - a data collection service[3]
- Gnocchi - a resource indexing and metric storage service[4]


-- what's with the name --
the generic name of Telemetry is not to assert authority over this 
space, we are just a collective who have interest and contribute to 
parts of this domain. each of the Aodh, Ceilometer, and Gnocchi teams 
are interested and open in collaborating[5] and building up this domain.



-- what happens next --
for now, as the projects are closely collaborating, they will share 
design and meeting space until they are deemed too large to share.


from a consumer pov, nothing should really change. the services are 
already packaged and separated into Aodh, Ceilometer, Gnocchi and the 
documentation already references Telemetry[6]


Going forward the teams will continue to improve each of the services[7] 
and help others extend them[5].


as always, anyone is welcome to contribute or provide feedback on each 
of the projects:

- irc: #openstack-telemetry on Freenode for questions
- ml: [telemetry] subject for general topics, [aodh], [ceilometer], 
or [gnocchi] for project specific topics



if there are any questions, please let us know so we can make transition 
go smoothly.



[1] https://wiki.openstack.org/wiki/Telemetry
[2] http://docs.openstack.org/developer/aodh
[3] http://docs.openstack.org/developer/ceilometer
[4] http://docs.openstack.org/developer/gnocchi
[5] https://wiki.openstack.org/wiki/Telemetry#Externally_Managed
[6] http://docs.openstack.org/admin-guide-cloud/telemetry.html
[7] 
http://lists.openstack.org/pipermail/openstack-dev/2015-November/078461.html


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] zombie process after ceilometer-api

2015-11-12 Thread gord chung

if you have configured two workers and you have two ceilometer-api 
processes and an extra 'zombie' process, then it's correct.


someone else might be able to articulate this better but when you use 
workers, it creates child processes that represent the workers you set, 
and there is a parent process which manages them.


you'll probably notice the same if you configure addition 
collector/notification agent workers... and the same for other projects


cheers,

On 12/11/15 09:26 AM, Madhu Mohan Nelemane wrote:
Here is the output of "ps aux | grep ceilometer-api" after "ceilometer 
meter-list"


ceilome+ 27192  1.0  1.1 247400 60388 ?Ss   14:10   0:00 
/usr/bin/python /usr/bin/ceilometer-api 
--config-file=/etc/ceilometer/ceilometer.conf 
--logfile=/var/log/ceilometer/api.log
ceilome+ 27655 15.3  0.0  0 0 ?Z14:11   0:00 
[ceilometer-api] 


Its the second line I am concerned about.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [telemetry] Bug opening policy

2015-11-12 Thread gord chung




On 12/11/2015 10:44 AM, Julien Danjou wrote:

Hi telemetry team and contributors,

We are starting to distinguish a terrible trend in our bug tracking
system. Some people tend to:

1. Open a bug for a (trivial) issue (e.g. a typo in the documentation)
2. Submit a patch 2 minutes later fixing the bug opened at step 1.

We'd like to emphasize that this useless and represents actually a waste
of time both for the bug opener and us Launchpad guardians.

We do highly appreciate contributions, in any form, bug opening or bug
fixing – we just don't require having a bug opened for each of our
change.

The bug tracking system is made for registering issues that you don't
have solutions for. If you already have the fix for it, just send the
fix – we'll just be glad and we'll review it!


or someone will ask you to open a bug depending on proximity to release.

but yes, please no typo bugs. feel free to forward this to managers -- 
my nickname is cdent on irc.*


* this is a joke.

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] zombie process after ceilometer-api

2015-11-12 Thread gord chung

can you clarify what you mean by 'zombie process'? i'm not aware of a 
bug relating to this so could you open one.


just for reference, this is the suggested method for deploying the 
ceilometer-api: 
http://docs.openstack.org/developer/ceilometer/install/mod_wsgi.html


cheers,

On 12/11/2015 8:37 AM, Madhu Mohan Nelemane wrote:

Hi,

I see a zombie process left out by ceilometer-api after a client 
command like "ceilometer meter-list".  This seems to occur only when I 
change the number of workers for [api] section in ceilometer.conf to 2 
or more.

With default  value of workers = 1, there is no zombie.

Has anybody encountered this problem ? OR is there already a bug to 
tackle this ?


Thanks,
Madhu Mohan


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [telemetry][gnocchi] defining scope/domain

2015-11-12 Thread gord chung




On 11/11/2015 4:19 AM, Julien Danjou wrote:

Yes, I think there is. We should write typical use cases and example in
the documentation so that people may get a grasp of what Gnocchi is and
what kind of problem it can or cannot solve.
cool stuff... how should we start this? etherpad[1]? i think we can 
publish it to main docs page as we come to agreement on each use case.


[1] https://etherpad.openstack.org/p/telemetry-gnocchi-use-cases

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] proposal to add Rohit Jaiswal to Ceilometer core

2015-11-11 Thread gord chung


thanks for the feedback.

it's my pleasure to welcome Rohit to the Ceilometer core team. everyone 
back to work! :)


On 05/11/15 08:45 AM, gord chung wrote:

hi folks,

i'd like to nominate Rohit Jaiswal as core for Ceilometer. he's done a 
lot of good work recently like discovering and fixing many issues with 
Events and implemented the configuration reloading functionality. he's 
also been very active providing input and fixes for many bugs.


as we've been doing, please vote here: 
https://review.openstack.org/#/c/242058/


reviews:
https://review.openstack.org/#/q/reviewer:%22Rohit+Jaiswal+%253Crohit.jaiswal%2540hp.com%253E%22+project:openstack/ceilometer,n,z 



patches:
https://review.openstack.org/#/q/owner:%22Rohit+Jaiswal+%253Crohit.jaiswal%2540hp.com%253E%22+project:openstack/ceilometer,n,z 

https://review.openstack.org/#/q/owner:%22Rohit+Jaiswal+%253Crohit.jaiswal%2540hp.com%253E%22+project:openstack/python-ceilometerclient,n,z 



cheers,



--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [telemetry][gnocchi] defining scope/domain

2015-11-10 Thread gord chung


hi,

i was doing some googling on the current state of time series databases 
to see if there was a worthwhile db to implement as a Gnocchi driver and 
it seems we're currently in the same state as before: there are flaws 
with all open source time series databases but if you have lots of 
money, Splunk is legit[1].


diving into each of the existing tsdb, there was a common theme that 
kept cropping up -- each of the tsdb claimed to do here> really, really well. by association, as Gnocchi captures time 
series data, it's assumed Gnocchi should do  
really, really well. this is obviously incorrect and setting Gnocchi up 
to fail.


as i get asked what Gnocchi is/does pretty often, i usually point them 
to docs[2]. that said, do we think it's necessary to define and list 
explicit use cases for Gnocchi? Gnocchi is 'a resource indexing and 
metric storage service' is a nice one-liner but would a comparison of 
Gnocchi make it easier to understand[3]. just wondering if there's a  
way we can make Gnocchi easier to understand and consumable.


[1] https://news.ycombinator.com/item?id=10530983
[2] http://docs.openstack.org/developer/gnocchi/
[3] http://prometheus.io/docs/introduction/comparison/

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] OpenStack Tokyo Summit Summary

2015-11-10 Thread gord chung

for reference, Julien is currently signed up for that task[1]. if this 
is a high priority requirement, we might want to see if another resource 
can pick it up, we can help out to explain how it should all be done :)


cheers,

[1] https://etherpad.openstack.org/p/mitaka-telemetry-todos

On 07/11/2015 2:09 PM, Fei Long Wang wrote:

Thanks, Doug. I will talk with Ceilometer team to see what we can do.

On 07/11/15 07:36, Doug Hellmann wrote:

Excerpts from Fei Long Wang's message of 2015-11-07 01:31:09 +1300:

Greetings,

Firstly, thank you for everyone joined Zaqar sessions at Tokyo summit.
We definitely made some great progress for those working sessions. Here
are the high level summary and those are basically our Mitaka
priorities. I may miss something so please feel free to comment/reply
this mail.

Sahara + Zaqar
-

We have a great discussion with Ethan Gafford from Sahara team. Sahara
team is happy to use Zaqar to fix some potential security issues. The
main user case will be covered in Mitaka is protecting tenant guest and
data from administrative user. So what Zaqar team needs to do in Mitaka
is completing the zaqar client function gaps for v2 to support signed
URL, which will be used by Sahara guest agent. Ethan will create a spec
in Sahara to track this work. This is a POC of what it'd look like to
have a guest agent in Sahara on top of Zaqar. The Sahara team has not
decided to use Zaqar yet but this would be the bases for that 
discussion.


Horizon + Zaqar
--

We used 1 horizon work session and 1 Zaqar work session to discuss this
topic. The main user case we would like to address is the async
notification so that Horizon won't have to poll the other OpenStack
components(e.g. Nova, Glance or Cinder) per second to get the latest
status. And I'm really happy to see we worked out a basic plan by
leveraging Zaqar's notification and websocket.

1. Implement a basic filter for Zaqar subscription, so that Zaqar can
decide if the message should be posted/forwarded to the subscriber when
there is a new message posted the queue. With this feature, Horizon 
will

only be notified by its interested notifications.

https://blueprints.launchpad.net/zaqar/+spec/suport-filter-for-subscription 



2. Listen OpenStack notifications

We may need more discussion about this to make sure if it should be in
the scope of Zaqar's services. It could be separated process/service of
Zaqar to listen/collect interested notifications/messages and post them
in to particular Zaqar queues. It sounds very interesting and useful 
but

we need to define the scope carefully for sure.

The Telemetry team discussed splitting out the code in Ceilometer that
listens for notifications to make it a more generic service that accepts
plugins to process events. One such plugin might be an interface to
filter events and republish them to Zaqar, so if you're interested in
working on that you might want to coordinate with the Telemetry team to
avoid duplicating effort.



Pool Group and Flavor
-

Thanks MD MADEEM proposed this topic so that we have a chance to review
the design of pool, pool group and flavor. Now the pool group and 
flavor

has a 1:1 mapping relationship and the pool group and pool has a 1:n
mapping relationship. But end user don't know the existence of pool, so
flavor is the way for end user to select what kind of storage(based on
capabilities) he want to use. Since pool group can't provide more
information than flavor so it's not really necessary, so we decide to
deprecate/remove it in Mitaka. Given this is hidden from users (done
automatically by Zaqar), there won't be an impact on the end user and
the API backwards compatibility will be kept.

https://blueprints.launchpad.net/zaqar/+spec/deprecate-pool-group

Zaqar Client


Some function gaps need to be filled in Mitaka. Personally, I would 
rate

the client work as the 1st priority of M since it's very key for the
integration with other OpenStack components. For v1.1, the support for
pool and flavor hasn't been completed. For v2, we're stilling missing
the support for subscription and signed URL.

https://blueprints.launchpad.net/zaqar/+spec/finish-client-support-for-v1.1-features 



SqlAlchemy Migration
-

Now we're missing the db migration support for SqlAlchemy, the control
plane driver. We will fix it in M as well.

https://blueprints.launchpad.net/zaqar/+spec/sqlalchemy-migration



Guys, please contribute this thread to fill the points/things I missed
or pop up in #openstack-zaqar channel directly with questions and
suggestions.

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listi

Re: [openstack-dev] [Ceilometer] [Bug#1497073]The return sample body of sample-list is different when use -m and not

2015-11-09 Thread gord chung

again, it's been debated that we shouldn't be allowed to completely drop 
apis.


in regards to deprecation, how do you make this deprecation known? also, 
this goes beyond the client. if we deprecated in the client we should 
really have the endpoint deprecated in API as well.


On 05/11/2015 8:25 PM, liusheng wrote:
I don't think we need two APIs to act duplicated functionalities, the 
"sample-list -m" command actually invoke API "GET 
/V2/meters/", it is more like a meter related API, not 
sample. I personally prefer to mark the "sample-list -m" command 
deprecated and dropped in future cycle. is this reasonable ?


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer]:Subscribe and Publish Notification frame work in Ceilometer !

2015-11-05 Thread gord chung




On 05/11/2015 5:11 AM, Raghunath D wrote:

Hi Pradeep,

Presently we are looking for a monitoring service.Using monitoring 
service user's/application's
will subscribe for few notification's/events from openstack 
infrastructure and monitoring service

will publish these notification to user's/application's.

We are exploring Ceilometer for this purpose.We came across below blue 
print which is similar to our requirement.


 https://blueprints.launchpad.net/ceilometer/+spec/declarative-notifications.


i'm not exactly clear on what you are trying to achieve. that said, the 
basic premise of the above blueprint is that if serviceX (nova, neutron, 
etc...) starts publishing a new notification with a metric of interest, 
Ceilometer can be easily configured to capture said metric by adding a 
metric definition to a definition file[1] or a custom definition 
file[2]. the same can be done for events[3].




We have few queries on declarative-notifications frame work,could you 
please help us in addressing them:


1.We are looking for an API for Subcribing/Publishing notification.Do 
this frame work exposes any such API,if yes could you

   please provide us API doc or spec how to use it.
2.If the frame work doesn't have such API,does any of the development 
group is working in this area.
3.Please suggest what would be the best place in ceilometer 
notification frame work(publisher/dispatcher/..)

   to implement the Subscribe and Publish API.


from what is described, it seems like you'd like Ceilometer to capture a 
notification and republish it rather than stored in a Ceilometer 
supported storage driver (ie Gnocchi, ElasticSearch, SQL, etc...). 
currently, the only way to do this is to not enable a collector service. 
doing so, the Event/Sample will be published to a message queue 
(default) which you can configure your service to pull from. currently, 
i don't believe oslo.messaging supports pub/sub work flow. 
alternatively, you can use one of the other publishers[4]. the kafka 
publisher should allow you to do a pub/sub type workflow. i know RAX has 
atom hopper[5] which uses atom feeds to support pub/sub functionality. 
there was discussions on adding support for this but no work has been 
done on it. feel free to propose it if you feel it's worthwhile.


[1] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml
[2] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/notifications.py#L31
[3] 
https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml
[4] 
http://docs.openstack.org/admin-guide-cloud/telemetry-data-retrieval.html#publishers

[5] http://atomhopper.org/

cheers,

--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] [Bug#1497073]The return sample body of sample-list is different when use -m and not

2015-11-05 Thread gord chung

i'm sort of torn on this item. there's a general feeling that regarding 
api, nothing should be dropped so i'm hesitant to actually deprecate it. 
i think changing the data also is very dangerous when it comes to 
compatibility (even though keeping it increases inconsistency).


maybe the better solution is to document that these are different APIs 
and will return different results.


On 05/11/2015 2:30 AM, Lin Juan IX Xia wrote:

Hi,

Here is an open bug : https://bugs.launchpad.net/ceilometer/+bug/1497073

Is it a bug or not?

For the command "ceilometer sample-list --meter cpu", it calls 
"/v2/meter" API and return the OldSample objects
which return body is different from "ceilometer sample-list --query 
'meter=cpu'".
To fix this inconformity, we can deprecate the command using -m or fix 
it to return the same body as command sample-list

Best Regards,
Xia Linjuan




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo_messaging] Regarding " WARNING [oslo_messaging.server] wait() should have been called after stop() as wait() ...

2015-11-05 Thread gord chung

my understanding is that if you are calling stop()/wait() your intention 
is to shut down the listener. if you intend on keeping an active 
consumer on the queue, you shouldn't be calling either stop() or wait(), 
just start.


On 05/11/2015 2:07 PM, Nader Lahouti wrote:


Thanks for the pointer, I'll look into it. But one question, by 
calling stop() and then wait(), does it mean the application has to 
call start() again after the wait()? to process more messages?


I am also using 
http://docs.openstack.org/developer/oslo.messaging/server.html for the 
RPC server

Does it mean there has to be stop() and then wait() there as well?


Thanks,
Nader.



On Thu, Nov 5, 2015 at 10:19 AM, gord chung <mailto:g...@live.ca>> wrote:




On 05/11/2015 1:06 PM, Nader Lahouti wrote:

Hi Doug,

I have an app that listens to notifications and used the info
provided in

http://docs.openstack.org/developer/oslo.messaging/notification_listener.html


Basically I create
1. NotificationEndpoints(object):

https://github.com/openstack/networking-cisco/blob/master/networking_cisco/apps/saf/common/rpc.py#L89
2. NotifcationListener(object):

https://github.com/openstack/networking-cisco/blob/master/networking_cisco/apps/saf/common/rpc.py#L100
3. and call start() and  then wait()


the correct usage is to call stop() before wait()[1]. for
reference on how to use listeners, you can see Ceilometer[2]


[1]http://docs.openstack.org/developer/oslo.messaging/notification_listener.html
[2]
https://github.com/openstack/ceilometer/blob/master/ceilometer/utils.py#L250

-- 
gord




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo_messaging] Regarding " WARNING [oslo_messaging.server] wait() should have been called after stop() as wait() ...

2015-11-05 Thread gord chung




On 05/11/2015 1:06 PM, Nader Lahouti wrote:

Hi Doug,

I have an app that listens to notifications and used the info provided in
http://docs.openstack.org/developer/oslo.messaging/notification_listener.html


Basically I create
1. NotificationEndpoints(object):
https://github.com/openstack/networking-cisco/blob/master/networking_cisco/apps/saf/common/rpc.py#L89
2. NotifcationListener(object):
https://github.com/openstack/networking-cisco/blob/master/networking_cisco/apps/saf/common/rpc.py#L100
3. and call start() and  then wait()


the correct usage is to call stop() before wait()[1]. for reference on 
how to use listeners, you can see Ceilometer[2]


[1]http://docs.openstack.org/developer/oslo.messaging/notification_listener.html
[2] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/utils.py#L250


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] proposal to add Rohit Jaiswal to Ceilometer core

2015-11-05 Thread gord chung


hi folks,

i'd like to nominate Rohit Jaiswal as core for Ceilometer. he's done a 
lot of good work recently like discovering and fixing many issues with 
Events and implemented the configuration reloading functionality. he's 
also been very active providing input and fixes for many bugs.


as we've been doing, please vote here: 
https://review.openstack.org/#/c/242058/


reviews:
https://review.openstack.org/#/q/reviewer:%22Rohit+Jaiswal+%253Crohit.jaiswal%2540hp.com%253E%22+project:openstack/ceilometer,n,z

patches:
https://review.openstack.org/#/q/owner:%22Rohit+Jaiswal+%253Crohit.jaiswal%2540hp.com%253E%22+project:openstack/ceilometer,n,z
https://review.openstack.org/#/q/owner:%22Rohit+Jaiswal+%253Crohit.jaiswal%2540hp.com%253E%22+project:openstack/python-ceilometerclient,n,z

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [telemetry][ceilometer][aodh][gnocchi] Tokyo summit roundup

2015-11-04 Thread gord chung


hi folks,

i want to start off by thanking everyone for joining the telemetry 
related discussions at the Tokyo summit -- we got some great feedback 
and ideas. similar to the last summit, here's a rundown of items that we 
talked about[1] and will be tracking in the upcoming cycle. as before, 
this is a (chaotic) brain dump and does not necessarily reflect any 
prioritisation.


note: the following is split into different sections, each representing 
a service provided under the telemetry umbrella. these projects are 
discretely managed but with tight collaboration.



-- Aodh (alarming service) --

- client - since we split alarming functionality from Ceilometer into 
it's own unique service[2], aodhclient support will be added so 
ceilometerclient functionality does not become overloaded with unrelated 
alarming code.
- user interface - to improve usability, support for CRUD operations of 
alarms will be added to horizon [3]
- horizontal scaling - the existing event alarm support added in 
Liberty[4] handles a single evaluator. multiple worker support will be 
added to enable better scaling
- simplifying combination alarm - combination alarms allowed flexibility 
of reusing threshold alarms but limited the use case to AND conditions 
and added evaluation ordering complexity. these drawbacks will be 
addressed by a new composite alarm [5]
- testing - tempest and grenade plugin testing support to be added in 
addition to existing unit/functional tests



-- Ceilometer (data collection service) --

- example reference architecture - to improve the consumption of 
Ceilometer, performance study will be done to build reference 
architecture. additional example configurations will be added to enable 
easier entry based on use case.
- housekeeping - alarming and rpc functionality were deprecated in 
Kilo/Liberty[6]. to ensure a tidy code base, it's time to clean house or 
as some devs like to put it: burn it down.[*]

- rolling upgrades - document the process of upgrading
- refined polling - the polling agent(s) now exclusively poll data and 
defer processing to notification agent(s). because of this, we can 
create a more tailored polling and pipeline configuration experience.
- improved polling distribution - currently, the cache is shared within 
a process. to better enable task distribution, we should enable sharing 
the cache across processes.
- polling metadata caching - we improved the caching mechanism in 
Liberty to minimise the load caused by Ceilometer polling. further 
improvements can be made to minimise the number of calls in general.[7]
- resource caching - to improve write speed, a cache will be implemented 
in the collector to minimise unnecessary writes[8]
- batch db writing - to minimise writes, batched writing of data will be 
added to collector[9]
- componentisation part 2 - Ceilometer handles meters[10] and 
events[11]. we need to make it pluggable to offer better flexibility.
- testing - tempest and grenade plugin testing support to be added in 
addition to existing unit/functional tests. additionally, multi-node 
testing to test upgrade path.



-- Gnocchi (time-series database and resource indexing service) --

- metric archive sharding - to improve performance of very large data 
sets (ie. second-by-second granularity), we can split the archive when 
updating data to avoid transferring entire archive set.
- dynamic resource creation - currently to create a new resource type, a 
new model and migration needs to be added. we need to make this creation 
dynamic to allow for more resource types.
- proliferate gnocchiclient adoption - gnocchiclient is now available. 
to ensure consistent usage, it should be adopted in Ceilometer and Aodh.
- testing - tempest and grenade plugin testing support to be added in 
addition to existing unit/functional tests



you can sign up for work items on the etherpad[12]. as always, you're 
more than welcome to propose addition ideas on irc:#openstack-ceilometer 
(to be #openstack-telemetry) and to openstack-dev using  
[telemetry]/[ceilometer]/[aodh]/[gnocchi] in subject.


as always we will continue to work with externally managed projects[13].

[1] 
https://wiki.openstack.org/wiki/Design_Summit/Mitaka/Etherpads#Ceilometer
[2] 
http://lists.openstack.org/pipermail/openstack-dev/2015-September/073897.html
[3] 
https://blueprints.launchpad.net/openstack-ux/+spec/horizon-alarm-management
[4] 
http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/event-alarm-evaluator.html

[5] https://review.openstack.org/#/c/208786/
[6] 
https://wiki.openstack.org/wiki/ReleaseNotes/Liberty#OpenStack_Telemetry_.28Ceilometer.29

[7] https://review.openstack.org/#/c/209799/
[8] https://review.openstack.org/#/c/203109/
[9] https://review.openstack.org/#/c/234831/
[10] http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html
[11] http://docs.openstack.org/admin-guide-cloud/telemetry-events.html
[12] https://etherpad.openstack.org/p/mitaka-telemetry-t

Re: [openstack-dev] [oslo][oslo.config][oslo.service] Dynamic Reconfiguration of OpenStack Services

2015-11-04 Thread gord chung

we actually had a solution implemented in Ceilometer to handle this[1].

that said, based on the results of our survey[2], we found that most
operators *never* update configuration files after the initial setup and
if they did it was very rarely (monthly updates). the question related
to Ceilometer and its pipeline configuration file so the results might
be specific to Ceilometer. I think you should definitely query operators
before undertaking any work. the last thing you want to do is implement
a feature no one really needs/wants.

[1]
http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/reload-file-based-pipeline-configuration.html
[2]
http://lists.openstack.org/pipermail/openstack-dev/2015-September/075628.html

On 04/11/2015 10:00 AM, Marian Horban wrote:

Hi guys,

Unfortunately I haven't been on Tokio summit but I know that there was
discussion about dynamic reloading of configuration.
Etherpad refs:
https://etherpad.openstack.org/p/mitaka-cross-project-dynamic-config-services,

https://etherpad.openstack.org/p/mitaka-oslo-security-logging

In this thread I want to discuss agreements reached on the summit and
discuss

implementation details.

Some notes taken from etherpad and my remarks:

1. "Adding "mutable" parameter for each option."
"Do we have an option mutable=True on CfgOpt? Yes"
-
As I understood 'mutable' parameter must indicate whether service
contains

code responsible for reloading of this option or not.
And this parameter should be one of the arguments of cfg.Opt constructor.
Problems:
1. Library's options.
SSL options ca_file, cert_file, key_file taken from oslo.service library
could be reloaded in nova-api so these options should be mutable...
But for some projects that don't need SSL support reloading of SSL
options

doesn't make sense. For such projects this option should be non mutable.
Problem is that oslo.service - single and there are many different
projects

which use it in different way.
The same options could be mutable and non mutable in different contexts.
2. Support of config options on some platforms.
Parameter "mutable" could be different for different platforms. Some
options
make sense only for specific platforms. If we mark such options as
mutable

it could be misleading on some platforms.
3. Dependency of options.
There are many 'workers' options(osapi_compute_workers, ec2_workers,
metadata_workers, workers). These options specify number of workers for
OpenStack API services.
If value of the 'workers' option is greater than '1' instance of
ProcessLauncher is created otherwise instance of ServiceLauncher is
created.

When ProcessLauncher receives SIGHUP it reloads it own configuration,
gracefully terminates children and respawns new children.
This mechanism allows to reload many config options implicitly.
But if value of the 'workers' option equals '1' instance of
ServiceLauncher

is created.
ServiceLauncher starts everything in single process and in this case we
don't have such implicit reloading.

I think that mutability of options is a complicated feature and I
think that
adding of 'mutable' parameter into cfg.Opt constructor could just add
mess.

2. "oslo.service catches SIGHUP and calls oslo.config"
-
From my point of view every service should register list of hooks to
reload
config options. oslo.service should catch SIGHUP and call list of
registered

hooks one by one with specified order.
Discussion of such implementation was started in ML:
http://lists.openstack.org/pipermail/openstack-dev/2015-September/074558.html.
Raw reviews:
https://review.openstack.org/#/c/228892/,
https://review.openstack.org/#/c/223668/.

3. "oslo.config is responsible to log changes which were ignored on
SIGHUP"

-
Some config options could be changed using API(for example quotas)
that's why

oslo.config doesn't know actual configuration of service and can't log
changes of configuration.

Regards, Marian Horban

--
gord

Re: [openstack-dev] [all][api][tc][perfromance] API for getting only status of resources

2015-11-04 Thread gord chung

apologies, if the below was mentioned at some point in this thread.

On 04/11/2015 10:42 AM, Sean Dague wrote:

This seems like a fundamental abuse of HTTP honestly. If you find
yourself creating a ton of new headers, you are probably doing it wrong.
if we want to explore the HTTP path, did we consider using ETags[1] to
check whether resources have changed? it's something used by Gnocchi's
API to handle resource changes.

I do think the near term work around is to actually use Searchlight.
They're monitoring the notifications bus for nova, and refreshing
resources when they see a notification which might have changed it. It
still means that Searchlight is hitting our API more than ideal, but at
least only one service is doing so, and if the rest hit that instead
they'll get the resource without any db hits (it's all through an
elastic search cluster).

I think longer term we probably need a dedicated event service in
OpenStack. A few of us actually had an informal conversation about this
during the Nova notifications session to figure out if there was a way
to optimize the Searchlight path. Nearly everyone wants websockets,
which is good. The problem is, that means you've got to anticipate
10,000+ open websockets as soon as we expose this. Which means the stack
to deliver that sanely isn't just a bit of python code, it's also the
highly optimized server underneath.
as part of the StackTach integration efforts, Ceilometer (as of Juno)
listens to all notifications in the OpenStack ecosystem and builds a
normalised event model[2] from it. the normalised event data is stored
in a backend (elasticsearch, sql, mongodb, hbase) and from this you can
query based on required attributes. in addition to storing events, in
Liberty, Aodh (alarming service) added support to take events and create
alarms based on change of state[3] with expanded functionality to be
added. this was added to handle the NFV use case but may also be
relevant here as it seems like we want to have an action based on status
changes.

i should mention that we discussed splitting out the event logic in
Ceilometer to create a generic listener[4] service which could convert
notification data to meters, events, and anything else. this isn't a
high priority item but might be an integration point for those looking
to leverage notifications in OpenStack.

[1] https://en.wikipedia.org/wiki/HTTP_ETag
[2] http://docs.openstack.org/admin-guide-cloud/telemetry-events.html
[3]
http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/event-alarm-evaluator.html

[4] https://etherpad.openstack.org/p/mitaka-telemetry-split

cheers,

--
gord

[openstack-dev] [telemetry][ceilometer][aodh][gnocchi] no (in person) mid-cycle for Mitaka

2015-11-04 Thread gord chung


hi all,

after discussions on the usefulness of a Telemetry mid-cycle, we've 
decided to forego having an in-person mid-cycle for Mitaka. to avoid 
rehashing already discussed items, see: same reasons as Glance[1]. much 
thanks to jasonamyers for offering a venue for a Telemetry mid-cycle.


that said we will try to have a virtual one similar to Liberty[2] should 
any items pop up over the development cycle. we would target some time 
during January.


looking forward to N*, an in-person mid-cycle might be beneficial if all 
data/telemetry related projects were to meetup. we had great 
participation from projects such as CloudKitty, Vitrage, etc...[3] which 
leverage parts of Ceilometer/Aodh/Gnocchi. if we all came together, i 
think that would make a worthwhile mid-cycle. this is something we can 
discuss over the coming cycle.


[1] 
http://lists.openstack.org/pipermail/openstack-dev/2015-November/078239.html

[2] http://lists.openstack.org/pipermail/openstack-dev/2015-July/068911.html
[3] https://wiki.openstack.org/wiki/Ceilometer#Ceilometer_Extensions

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer][aodh] proposal to add Ryota Mibu to Aodh core

2015-10-21 Thread gord chung


thanks for the feedback folks.

i'd like to welcome Ryota to the Aodh core team. thank you for all the 
continued effort in building alarming functionality in OpenStack!


On 16/10/2015 8:59 AM, gord chung wrote:
sorry, as we usually do, please comment on gerrit[1]. if you cannot, 
then please comment here.


[1] https://review.openstack.org/#/c/235890/

On 16/10/15 08:39 AM, gord chung wrote:

hi,

i'd like to nominate Ryota Mibu to the Aodh core team. he has been an 
active participant in Aodh and leads the Event alarms work done 
there. he also brings good insight related to NFV use case.


patches:
https://review.openstack.org/#/q/owner:%22Ryota+MIBU%22+project:openstack/aodh,n,z 



reviews:
https://review.openstack.org/#/q/reviewer:%22Ryota+MIBU%22+project:openstack/aodh,n,z 



cheers,





--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Meeting cancelled for this week

2015-10-21 Thread gord chung


thanks Ildiko

it's also safe to assume the meeting will be cancelled for October 29 
and November 5. please use the mailing list (it's actually preferred) 
for any discussion items.


On 21/10/2015 8:22 AM, Ildikó Váncsa wrote:

Hi All,

I would like to inform you that we cancel the Ceilometer meeting for this week 
in favor of Summit travels and preparation.

We finished to organize our sessions for next week, you can find the schedule 
here: https://mitakadesignsummit.sched.org/overview/type/ceilometer
The etherpads for the sessions are available here: 
https://wiki.openstack.org/wiki/Design_Summit/Mitaka/Etherpads#Ceilometer

Best Regards,
Ildikó

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] [watcher] New project: Watcher

2015-10-21 Thread gord chung


hi,

seems like a neat idea, i had a few questions after watching the 
presentation from Vancouver.


- is Watcher streaming the data captured by Ceilometer or is it querying 
the Ceilometer db?
- is it utilising the Event data captured by Ceilometer or metric data? 
both?

- i notice there's a tsdb used by Watcher. what TSDB is supported?
- is alarming functionality provided by Aodh used or is the 'machine 
learning'+'orchestration' service meant to replace the functionality 
provided by Aodh+Heat?


one reason i'm asking is that Ceilometer has a section in wiki to list 
projects that leverage Ceilometer[1] so it's easy to track existing 
extensions. feel free to add Watcher there if you feel like it.


[1] https://wiki.openstack.org/wiki/Ceilometer#Ceilometer_Extensions

cheers,

On 21/10/2015 8:22 AM, Antoine CABOT wrote:

We are pleased to introduce Watcher, a new project in the OpenStack
ecosystem. We believe that a "resource optimization" service in an
OpenStack-based cloud is a crucial component that has been missing to
date and we have started working towards filling that gap.

OpenStack Watcher provides a flexible and scalable resource
optimization service for multi-tenant OpenStack-based clouds.
Watcher provides a complete optimization loop—including everything
from a metrics receiver, complex event processor and profiler,
optimization processor and an action plan applier. This provides a
robust framework to realize a wide range of cloud optimization goals,
including the reduction of data center operating costs, increased
system performance via intelligent virtual machine migration,
increased energy efficiency—and more!

Not only does Watcher provide several out-of-box optimization routines
for immediate value-add, but it also supports a pluggable
architecture by which custom optimization algorithms, data metrics and
data profilers can be developed and inserted into the Watcher framework.
Additionally, Watcher enables two different modes of execution—advise
mode (manual) or active mode (automatic), giving cloud administrators
the runtime flexibilities that their clouds require. Most importantly,
administrators of OpenStack-based clouds equipped with Watcher will
decrease their Total Cost of Ownership (TCO) by way of more efficient
use of their infrastructure and less “hands on” (read: manual)
administrator involvement to perform optimizations.

More information, including key use cases, architecture, etc., are
described on the project wiki [0].  Also feel free to browse some
source code [1, 2].

We meet weekly on Wednesdays at 16:00 UTC in the #openstack-meeting-3
IRC channel. You can also use #openstack-watcher to contact the team.

We hope to see you at our unconference session at the upcoming
OpenStack Summit in Tokyo and we are welcoming participation from the
community.  Let us know if you are interested in this topic and
join us in #openstack-watcher!

Regards,

Antoine Cabot (Watcher PTL) (acabot)
Susanne Balle (sballe)
Joe Cropper (jwcroppe)

[0] https://wiki.openstack.org/wiki/Watcher
[1] https://github.com/openstack/watcher
[2] https://github.com/openstack/python-watcherclient
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] "Infinite" values for "sum" and "average" fields in statistics query

2015-10-20 Thread gord chung

could you open a bug on 
https://bugs.launchpad.net/ceilometer/+bugs?orderby=-id&start=0 and 
reference this thread


i would also check to see if ceilometer-api logs have any noticeable 
warning/errors. it's possible a datatype is being restricted there as well.


thanks

On 20/10/2015 12:22 PM, Srikanth Vavilapalli wrote:

Hi

I am using MongoDB backend.

Thanks
Srikanth

-Original Message-----
From: gord chung [mailto:g...@live.ca]
Sent: Tuesday, October 20, 2015 9:20 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Ceilometer] "Infinite" values for "sum" and 
"average" fields in statistics query

which backend are you using? this potentially may be a bug where we're hitting 
some size limit restricted by datatype.

On 19/10/2015 4:02 PM, Srikanth Vavilapalli wrote:

Hi

I have observed "inf" values for "sum" and "average" fields in a ceilometer statistics query on one of my custom meter 
as shown below. I have also verified the samples query for this meter and there are 1544 samples (output shown below) with all of them having value 
as "150.0". So ideally the "Avg" filed should be "150.0" in this scenario. Could anyone tell me in what scenario we 
will see these two fields as "inf" in the meter statistics query? Plz let me know if you need more details or logs here. Appreciate your 
help.

Thanks
Srikanth


svavilap@ctl:/usr/lib/python2.7/dist-packages/ceilometer$ ceilometer
statistics --meter=vcpe.dns.cache.size
++++---+---+-+-+---++++
| Period | Period Start   | Period End | Max   | 
Min   | Avg | Sum | Count | Duration   | Duration Start | Duration 
End   |
++++---+---+-+-+---++++
| 0  | 2015-10-17T01:30:24.999000 | 2015-10-19T17:47:20.587000 | 150.0 | 
150.0 | inf | inf | 1544  | 231415.588 | 2015-10-17T01:30:24.999000 | 
2015-10-19T17:47:20.587000 |
++++---+---+-+-+---++++


svavilap@ctl:/usr/lib/python2.7/dist-packages/ceilometer$ ceilometer
sample-list --meter=vcpe.dns.cache.size
+-+-+---++-++
| Resource ID | Name| Type  | Volume | Unit| Timestamp  
|
+-+-+---++-++
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:47:20.587000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:47:00.048000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:42:20.47 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:41:59.921000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:37:20.362000 |
.
.
.
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-17T01:50:25.499000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-17T01:45:46.255000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-17T01:45:25.374000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-17T01:40:46.131000 |

__
 OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] "Infinite" values for "sum" and "average" fields in statistics query

2015-10-20 Thread gord chung

which backend are you using? this potentially may be a bug where we're 
hitting some size limit restricted by datatype.


On 19/10/2015 4:02 PM, Srikanth Vavilapalli wrote:

Hi

I have observed "inf" values for "sum" and "average" fields in a ceilometer statistics query on one of my custom meter 
as shown below. I have also verified the samples query for this meter and there are 1544 samples (output shown below) with all of them having value 
as "150.0". So ideally the "Avg" filed should be "150.0" in this scenario. Could anyone tell me in what scenario we 
will see these two fields as "inf" in the meter statistics query? Plz let me know if you need more details or logs here. Appreciate your 
help.

Thanks
Srikanth


svavilap@ctl:/usr/lib/python2.7/dist-packages/ceilometer$ ceilometer statistics 
--meter=vcpe.dns.cache.size
++++---+---+-+-+---++++
| Period | Period Start   | Period End | Max   | 
Min   | Avg | Sum | Count | Duration   | Duration Start | Duration 
End   |
++++---+---+-+-+---++++
| 0  | 2015-10-17T01:30:24.999000 | 2015-10-19T17:47:20.587000 | 150.0 | 
150.0 | inf | inf | 1544  | 231415.588 | 2015-10-17T01:30:24.999000 | 
2015-10-19T17:47:20.587000 |
++++---+---+-+-+---++++


svavilap@ctl:/usr/lib/python2.7/dist-packages/ceilometer$ ceilometer 
sample-list --meter=vcpe.dns.cache.size
+-+-+---++-++
| Resource ID | Name| Type  | Volume | Unit| Timestamp  
|
+-+-+---++-++
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:47:20.587000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:47:00.048000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:42:20.47 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:41:59.921000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-19T17:37:20.362000 |
.
.
.
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-17T01:50:25.499000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-17T01:45:46.255000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-17T01:45:25.374000 |
| vcpe-170| vcpe.dns.cache.size | gauge | 150.0  | entries | 
2015-10-17T01:40:46.131000 |

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] ceilometer+ops session

2015-10-19 Thread gord chung


hi folks,

to followup on an item regarding Ceilometer+ops session at the summit. 
there isn't one explicitly for Ceilometer but there is a monitoring 
session[1] and a billing session[2]. based on the past, these sessions 
tend to focus more on 3rd party tools such as Sensu et al but if free it 
might be worth your time to join as well.


as discussed, we'll be using our ceilometer contributor meet up time to 
discussion the results and feedback from surveys.


[1] 
https://mitakadesignsummit.sched.org/event/2b7a39ba370a9c9f7ab8b7de57ca6188
[2] 
https://mitakadesignsummit.sched.org/event/f72ce4bf2d403aec3357b3af2492ead2


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer][aodh] proposal to add Ryota Mibu to Aodh core

2015-10-16 Thread gord chung

sorry, as we usually do, please comment on gerrit[1]. if you cannot, 
then please comment here.


[1] https://review.openstack.org/#/c/235890/

On 16/10/15 08:39 AM, gord chung wrote:

hi,

i'd like to nominate Ryota Mibu to the Aodh core team. he has been an 
active participant in Aodh and leads the Event alarms work done there. 
he also brings good insight related to NFV use case.


patches:
https://review.openstack.org/#/q/owner:%22Ryota+MIBU%22+project:openstack/aodh,n,z 



reviews:
https://review.openstack.org/#/q/reviewer:%22Ryota+MIBU%22+project:openstack/aodh,n,z 



cheers,



--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer][aodh] proposal to add Ryota Mibu to Aodh core

2015-10-16 Thread gord chung


hi,

i'd like to nominate Ryota Mibu to the Aodh core team. he has been an 
active participant in Aodh and leads the Event alarms work done there. 
he also brings good insight related to NFV use case.


patches:
https://review.openstack.org/#/q/owner:%22Ryota+MIBU%22+project:openstack/aodh,n,z

reviews:
https://review.openstack.org/#/q/reviewer:%22Ryota+MIBU%22+project:openstack/aodh,n,z

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer][horizon] tentative design summit schedule

2015-10-13 Thread gord chung


hi,

for those interested, i've put up the tentative schedule[1] for 
telemetry-related topics for the Tokyo design summit.


note, there's a session to discuss horizon and ceilometer integration as 
that was a common issue raised in the recent ceilometer survey.


please let me know if there's any conflicts to work around.

[1] https://mitakadesignsummit.sched.org/overview/type/ceilometer

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] external projects extending ceilometer

2015-10-06 Thread gord chung


hi,

telemetry is a big space and its requirements differ from company to 
company. there are existing projects that extend/leverage the 
functionality of Ceilometer to either customise, fill in gaps or resolve 
complementary problems.


as the projects are not all managed by the Ceilometer team, i've added a 
section in the wiki[1] so there's a place for users to see what 
extensions exists and for developers to promote their customisations. 
feel free to add your own projects.


[1] https://wiki.openstack.org/wiki/Ceilometer#Ceilometer_Extensions

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer][aodh][gnocchi] Tokyo design session planning

2015-10-01 Thread gord chung


hi,

just a reminder, please submit any topics ASAP so we can properly vote 
on items. if you have a feature you'd like to submit this cycle it'd 
greatly help your chances of merging should you present/explain your 
topic to us common folks.


On 10/09/15 02:03 PM, gord chung wrote:

hi,

as mentioned during today's meeting, since we have our slots for 
design summit, we'll start accepting proposals for the 
telemetry-related topics for the Tokyo summit.


similar to previous design summits, anyone is welcome to propose 
topics related to ceilometer, aodh, gnocchi, or any other 
monitoring/metering/alarming related project.


official proposals can be submitted here: 
https://docs.google.com/spreadsheets/d/1ea_P2k1kNy_SILEnC-5Zl61IFhqEOrlZ5L8VY4-xNB0/edit?usp=sharing


we've also created an etherpad for those wishing to brainstorm ideas 
before adding formal proposal: 
https://etherpad.openstack.org/p/tokyo-ceilometer-design-summit


we have tentatively set submission deadline for October 5, 2015, which 
will be followed by a public vote.


cheers,
--
gord


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] OpenStack Telemetry user survey

2015-09-28 Thread gord chung


Hello,

The OpenStack Telemetry (aka Ceilometer) team would like to collect 
feedback and information from its user base in order to drive future 
improvements to the project.  To do so, we have developed a survey. It 
should take about 15min to complete.
Questions are fairly technical, so please ensure that you ask someone 
within your organization that is hands on using Ceilometer.


https://goo.gl/rKNhM1

On behalf of the Ceilometer community, we thank you for the time you 
will spend in helping us understand your needs.


--
Gordon Chung
Ceilometer PTL

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] ceilometer code debugging

2015-09-16 Thread gord chung


hi,

i'm not familiar with eclipse+pydev but you should be able to run that 
command as is in terminal


'./tools/ceilometer-test-event.py'

the main question i have is what are you trying to debug? that script, 
to be honest, does not do much. it seems to just test the event 
conversion functionality. you might find some useful information in the 
dev docs[1] if you have questions about Ceilometer. if it's a pydev 
specific question you might want to generalise your subject line to have 
more eyes.


[1] http://docs.openstack.org/developer/ceilometer


On 16/09/15 08:42 AM, Nanda Devi Sahu wrote:

Hello,

I am trying to debug Ceilometer code taken from git. I have configured 
Eclipse with Pydev for the same. The configuration seems to be fine 
but when I do a debug of the code it shows to be running but I do not 
see the flow of the code. LIke the main thread and the following threads.


Attached is the debug prespective view where the run is working 
without any code stack.


Could you please suggest what Am I missing to get the flow. I have 
also attached the debug configuration image for reference.



Regards,
Nanda.

*L&T Technology Services Ltd*

www.LntTechservices.com 

This Email may contain confidential or privileged information for the 
intended recipient (s). If you are not the intended recipient, please 
do not use or disseminate the information, notify the sender and 
delete it from your system.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] how to get current memory utilization of an instance

2015-09-16 Thread gord chung

is this using KVM? there are actually a few requirements[1] to ensure
memory.usage meter works using libvirt:

- libvirt 1.1.1+
- qemu 1.5+
- guest driver that supports memory balloon stats

retagging to ceilometer to get more eyes.

[1]
https://github.com/openstack/ceilometer-specs/blob/master/specs/kilo/libvirt-memory-utilization-inspector.rst#dependencies

cheers,

On 16/09/2015 6:23 AM, Abhishek Talwar wrote:

Hi Folks,

I have an OpenStack kilo setup and I am trying to get the memory usage
of the instances created in it.

I researched and found that ceilometer has a meter called
"memory.usage" that can give us the memory utilization of an instance.
However, on doing ceilometer meter-list it does not list
"memory.usage" as a meter. After searching I found that this problem
is being faced by many contributors and a possible solution is
changing nova configurations that does not help.

So can you help me with some other possible solution that can help me
in finding an instance's current memory utilization. I am working with
something that would require getting the memory and disk utilization.

Please provide a solution for this.

Thanks and Regards

Abhishek Talwar

=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

--
gord

[openstack-dev] [ceilometer] PTL candidacy

2015-09-15 Thread gord chung


hi folks,

less than six months ago, i decided to run for PTL of Ceilometer where 
my main goal was to support the community of contributors that exists 
within OpenStack with interests in telemetry[1]. it is under that tenet 
which i will run again for team lead of Ceilometer. as mentioned 
previously, we have a diverse set of contributors from across the globe 
working on various aspects of metering and monitoring and it is my goal 
to ensure nothing slows them down (myself included).


that said, as we look forward to Mitaka, i hope to follow along the path 
of stability, simplicity and usability. some items i'd like to target are:


- rolling upgrades - having a fluid upgrade path for operators is 
critical to providing a highly available cloud environment for their 
users. i would like to have a viable solution in Ceilometer that can 
provide this functionality with zero/minimal performance degradation.


- building up events - we started work on adding inline event alarming 
in Aodh during Liberty, this is something i'd like to improve upon by 
adding multiple worker support and broader alarming evaluations. also, a 
common use case for events is to analyse the data for BI. while we allow 
the ability to query and alarm on events. one useful tool would be the 
ability to run statistics on events such as the number of instances 
launched.


- optimising collection - we improved ease of use by adding declarative 
notification support in Liberty. it'd be great if this work could be 
adopted by projects producing metrics. additionally, we currently have a 
extremely tight coupling between resource metadata and measurement data. 
i'd like to evaluate how to loosen this so our data collection and 
storage is more flexible.


- continuing the refactoring - removing deprecated/redundant 
functionality; it was nice deprecating/deleting stuff, let's keep doing 
it (within reason)! one possible target would be splitting storage/api, 
while starting the initial deprecation of v2 metering api.


- functional and integration testing -  we now have integration and 
functional test living within our repositories. this should allow us to 
develop testing easier so it'd be good to broaden the coverage.


[1] 
http://lists.openstack.org/pipermail/openstack-dev/2015-April/060536.html


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] Ceilometer M Midcycle

2015-09-15 Thread gord chung


thanks for organising this Jason.

for those who can't attend but want to, i think it'd also be good to 
know if location is the blocker here.


retagging with [ceilometer].

On 15/09/2015 9:50 AM, Jason Myers wrote:

Hello Everyone,
We are setting up a few polls to determine the possibility of 
meeting face to face for a ceilometer midcycle in Dublin, IE. We'd 
like to gather for three days to discuss all the work we are currently 
doing; however, we have access to space for 5 so you could also use 
that space for co working outside of the meeting dates.  We have two 
date polls: one for Nov 30-Dec 18 at 
http://doodle.com/poll/hmukqwzvq7b54cef, and one for Jan 11-22 at 
http://doodle.com/poll/kbkmk5v2vass249i. You can vote for any of the 
days in there that work for you.  If we don't get enough interest in 
either poll, we will do a virtual midcycle like we did last year.  
Please vote for your favorite days in the two polls if you are 
interested in attending in person. If we don't get many votes, we'll 
circulate another poll for the virtual dates.


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

2015-09-15 Thread gord chung




On 03/09/2015 4:02 PM, Dan Smith wrote:

- do we need to migrate the db to some handle different set of
>attributes and what happens for nosql dbs?

No, Nova made no schema changes as a result of moving to objects.

so i don't really understand how this part works. if i have two 
collectors -- one collector writes v1 schema, one writes v2 schema -- 
how do they both write to the same db without anything changing in the 
db? presumably the db would only be configured to know how to store only 
one version?


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][TC] 'team:danger-not-diverse tag' and my concerns

2015-09-11 Thread gord chung




On 11/09/2015 3:26 PM, Joshua Harlow wrote:

Hi all,

I was reading over the TC IRC logs for this week (my weekly reading) 
and I just wanted to let my thoughts and comments be known on:


http://eavesdrop.openstack.org/meetings/tc/2015/tc.2015-09-08-20.01.log.html#l-309 



I feel it's very important to send a positive note for new/upcoming 
projects and libraries... (and for everyone to remember that most 
projects do start off with a small set of backers). So I just wanted 
to try to ensure that we send a positive note with any tag like this 
that gets created and applied and that we all (especially the TC) 
really really considers the negative connotations of applying that tag 
to a project (it may effectively ~kill~ that project).


I would really appreciate that instead of just applying this tag (or 
other similarly named tag to projects) that instead the TC try to 
actually help out projects with those potential tags in the first 
place (say perhaps by actively listing projects that may need more 
contributors from a variety of companies on the openstack blog under 
say a 'HELP WANTED' page or something). I'd much rather have that vs. 
any said tags, because the latter actually tries to help projects, vs 
just stamping them with a 'you are bad, figure out how to fix 
yourself, because you are not diverse' tag.


I believe it is the TC job (in part) to help make the community 
better, and not via tags like this that IMHO actually make it worse; I 
really hope that folks on the TC can look back at their own projects 
they may have created and ask how would their own project have turned 
out if they were stamped with a similar tag...


completely agree with everything here... i made a comment on the 
patch[1] regarding this and was told the idea was that the purpose of 
the tag was to note the potential fragility of a project if the leading 
company were to decide to pull out. this seems like a valid item to 
track but with that said, the existing wording of the proposal is not that.


[1] https://review.openstack.org/#/c/218725/

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometerclient] Updating global-requirements caps.

2015-09-11 Thread gord chung




On 10/09/2015 7:26 PM, Tony Breeds wrote:

Hi all,
 In trying to fix a few stable/juno issues we need to release a new version
of ceilometerclient for stable/juno.  This email is to try and raise awareness
so that if the proposal is bonkers [1] we can come up with something better.

This isn't currently possible due to the current caps in juno and kilo.

The proposed fix is to:

. update g-r in master (liberty): python-ceilometerclient>=1.2
   https://review.openstack.org/#/c/222386/
. update g-r in stable/kilo: python-ceilometerclient>=1.1.1,<1.2
. release a sync of stable/kilo g-r to stable/kilo python-ceilometerclient as 
1.1.1
. update g-r in stable/juno: python-ceilometerclient<1.1.0,!=1.0.13,!=1.0.14
. release 1.0.15 with a sync of stable/juno g-r

The point is, leave 1.0.x for juno, 1.1.x for kilo and >=1.2 for liberty

This is being tracked as: 
https://bugs.launchpad.net/python-ceilometerclient/+bug/1494516

There is a secondary issue if getting the (juno) gate in a shape where we can
actually do all of that.

Yours Tony.
[1] Bonkers is a recognized technical term right?


i commented on patch already but to reiterate, this sounds sane to me. 
we tagged stuff improperly during juno/kilo timespan so our versioning 
became an issue[1] and it looks like it caught up to us.


as it stands, version 1.1.0 is the rough equivalent to 1.0.14 (but with 
a requirement updates).  this seems to solve all the requirements issues 
so i'm content with the solution. thanks to both of you for figuring out 
the requirements logistics.


[1] 
http://eavesdrop.openstack.org/irclogs/%23openstack-ceilometer/%23openstack-ceilometer.2015-04-14.log.html#t2015-04-14T21:50:49


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer][aodh][gnocchi] Tokyo design session planning

2015-09-10 Thread gord chung


hi,

as mentioned during today's meeting, since we have our slots for design 
summit, we'll start accepting proposals for the telemetry-related topics 
for the Tokyo summit.


similar to previous design summits, anyone is welcome to propose topics 
related to ceilometer, aodh, gnocchi, or any other 
monitoring/metering/alarming related project.


official proposals can be submitted here: 
https://docs.google.com/spreadsheets/d/1ea_P2k1kNy_SILEnC-5Zl61IFhqEOrlZ5L8VY4-xNB0/edit?usp=sharing


we've also created an etherpad for those wishing to brainstorm ideas 
before adding formal proposal: 
https://etherpad.openstack.org/p/tokyo-ceilometer-design-summit


we have tentatively set submission deadline for October 5, 2015, which 
will be followed by a public vote.


cheers,

--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Getting Started : OpenStack

2015-09-08 Thread gord chung




On 07/09/2015 1:59 PM, Bhagyashree Uday wrote:

Hi ,

I am Bhagyashree from India(IRC nick : bee2502 ). I have previous 
experience in data analytics including Machine Leraning,,NLP,IR and 
User Experience Research. I am interested in contributing to OpenStack 
on projects involving data analysis. Also , if these projects could be 
a part of Outreachy, it would be added bonus. I went through project 
ideas listed on https://wiki.openstack.org/wiki/Internship_ideas and 
one of these projects interested me a lot -
Understand OpenStack Operations via Insights from Logs and Metrics: A 
Data Science Perspective
However, this project does not have any mentioned mentor and I was 
hoping you could provide me with some individual contact from 
OpenStack community who would be interested in mentoring this project 
or some mailing list/thread/IRC community where I could look for a 
mentor. Other open data science projects/idea suggestions are also 
welcome.




there was a project proposed a few months back called Cognitive[1] but i 
don't know the status of this project. as for Ceilometer, it doesn't 
encompass data analysis but it does collect data which you might be 
interested in leveraging (ie. resource metrics and system events)


[1] http://lists.openstack.org/pipermail/openstack-dev/2015-May/064195.html

--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Base feature deprecation policy

2015-09-08 Thread gord chung




On 08/09/2015 5:29 PM, Sean Dague wrote:

On 09/08/2015 03:32 PM, Doug Hellmann wrote:

Excerpts from Sean Dague's message of 2015-09-08 14:11:48 -0400:

On 09/08/2015 01:07 PM, Doug Hellmann wrote:

Excerpts from Dean Troyer's message of 2015-09-08 11:20:47 -0500:

On Tue, Sep 8, 2015 at 9:10 AM, Doug Hellmann

I'd like to come up with some way to express the time other than
N+M because in the middle of a cycle it can be confusing to know
what that means (if I want to deprecate something in August am I
far enough through the current cycle that it doesn't count?).

Also, as we start moving more projects to doing intermediate releases
the notion of a "release" vs. a "cycle" will drift apart, so we
want to talk about "stable releases" not just any old release.


I've always thought the appropriate equivalent for projects not following
the (old) integrated release cadence was for N == six months.  It sets
approx. the same pace and expectation with users/deployers.

For those deployments tracking trunk, a similar approach can be taken, in
that deprecating a config option in M3 then removing it in N1 might be too
quick, but rather wait at least the same point in the following release
cycle to increment 'N'.

dt


Making it explicitly date-based would simplify tracking, to be sure.

I would agree that the M3 -> N0 drop can be pretty quick, it can be 6
weeks (which I've seen happen). However N == six months might make FFE
deprecation lands in one release run into FFE in the next. For the CD
case my suggestion is > 3 months. Because if you aren't CDing in
increments smaller than that, and hence seeing the deprecation, you
aren't really doing the C part of CDing.

 -Sean


Do those 3 months need to span more than one stable release? For
projects doing intermediary releases, there may be several releases
within a 3 month period.

Yes. 1 stable release branch AND 3 months linear time is what I'd
consider reasonable.

-Sean

while the pyro in me would like to burn things asap, my fellow 
contributors won't let me so Ceilometer typically has done 
deprecate->deprecate->remove. but agree with sdague, the bare minimum 
should be ^. operators will yell, don't make them yell.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [aodh][ceilometer] (re)introducing Aodh - OpenStack Alarming

2015-09-08 Thread gord chung


hi all,

as you may have heard, in an effort to simplify OpenStack Telemetry 
(Ceilometer) and streamline it's code, the alarming functionality 
provided by OpenStack Telemetry has been moved to it's own 
repository[1]. The new project is called Aodh[2]. the idea is that Aodh 
will grow as it's own entity, with it's own distinct core team, under 
the Telemetry umbrella. this way, we will have a focused team 
specifically for the alarming aspects of Telemetry. as always, feedback 
and contributions are welcomed[3].


in the coming days, we will release a migration/changes document to 
explain the differences between the original alarming code and Aodh. all 
effort was made to maintain compatibility in configurations such that it 
should be possible to take the existing configuration and reuse it for 
Aodh deployment.


some quick notes:
- the existing alarming code will remain consumable for Liberty release 
(but in deprecated state)
- all new functionality (ie. inline/streaming alarm evaluations) will be 
added only to Aodh
- client and api support has been added to common Ceilometer interfaces 
such that if Aodh is enabled, the client can still be used and redirect 
to Aodh.

- mailing list items can be tagged with [aodh]
- irc discussions will remain under #openstack-ceilometer

many thanks for all those who worked on the code split and integration 
testing.


[1] https://github.com/openstack/aodh
[2] http://www.behindthename.com/name/aodh
[3] https://launchpad.net/aodh

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Meters

2015-09-08 Thread gord chung

is this using libvirt? if so, can you verify you have the following 
requirements:


 * libvirt 1.1.1+
 * qemu 1.5+
 * guest driver that supports memory balloon stats

also, please check to see if there are any visible ERRORs in 
ceilometer-agent-compute log.



On 08/09/2015 2:04 AM, Abhishek Talwar wrote:

Hi Folks,


I have installed a *kilo devstack setup* install and I am trying to 
get the *memory and disk usage* for my VM's. But on checking the 
*"ceilometer meter-list"* I can't find memory.usage or disk.usage meters.


I am searched a lot for this and still couldn't find a solution. So 
how to enable these meters to our meter-list.


I want all these meters in the ceilometer meter-list so that I can use 
them to monitor my instances.


|Currently the output of *ceilometer meter-list* is as follows:

|
|+--++--+--+--+
|Name|Type|Resource ID |User ID |Project ID |
+--++--+--+--+
| cpu | cumulative 
|5314c72b-a2b4-4b2b-bcb1-4057c3d96f77|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| cpu_util | gauge 
|5314c72b-a2b4-4b2b-bcb1-4057c3d96f77|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| disk.read.bytes | cumulative 
|5314c72b-a2b4-4b2b-bcb1-4057c3d96f77|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| disk.read.requests | cumulative 
|5314c72b-a2b4-4b2b-bcb1-4057c3d96f77|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| disk.write.bytes | cumulative 
|5314c72b-a2b4-4b2b-bcb1-4057c3d96f77|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| disk.write.requests | cumulative 
|5314c72b-a2b4-4b2b-bcb1-4057c3d96f77|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| image | gauge 
|55a0a2c2-8cfb-4882-ad05-01d7c821b1de||22f22fb60bf8496cb60e8498d93d56e8|
| image | gauge 
| acd6beef-13e6-4d64-a83d-9e96beac26ef||22f22fb60bf8496cb60e8498d93d56e8|
| image | gauge | ecefcd31-ae47-4079-bd19-efe07f4c33d3 
||22f22fb60bf8496cb60e8498d93d56e8|
| image.download | delta 
|55a0a2c2-8cfb-4882-ad05-01d7c821b1de||22f22fb60bf8496cb60e8498d93d56e8|
| image.serve | delta 
|55a0a2c2-8cfb-4882-ad05-01d7c821b1de||22f22fb60bf8496cb60e8498d93d56e8|
| image.size | gauge 
|55a0a2c2-8cfb-4882-ad05-01d7c821b1de||22f22fb60bf8496cb60e8498d93d56e8|
| image.size | gauge 
| acd6beef-13e6-4d64-a83d-9e96beac26ef||22f22fb60bf8496cb60e8498d93d56e8|
| image.size | gauge | ecefcd31-ae47-4079-bd19-efe07f4c33d3 
||22f22fb60bf8496cb60e8498d93d56e8|
| image.update | delta 
|55a0a2c2-8cfb-4882-ad05-01d7c821b1de||22f22fb60bf8496cb60e8498d93d56e8|
| image.upload | delta 
|55a0a2c2-8cfb-4882-ad05-01d7c821b1de||22f22fb60bf8496cb60e8498d93d56e8|
| instance | gauge 
|5314c72b-a2b4-4b2b-bcb1-4057c3d96f77|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| instance:m1.small | gauge 
|5314c72b-a2b4-4b2b-bcb1-4057c3d96f77|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| network.incoming.bytes | cumulative 
| nova-instance-instance-0022-fa163e3bd74e 
|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| network.incoming.packets | cumulative 
| nova-instance-instance-0022-fa163e3bd74e 
|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| network.outgoing.bytes | cumulative 
| nova-instance-instance-0022-fa163e3bd74e 
|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|
| network.outgoing.packets | cumulative 
| nova-instance-instance-0022-fa163e3bd74e 
|92876a1aad3c477398137b702a8467d3|22f22fb60bf8496cb60e8498d93d56e8|

+--++--+--+--+

Thanks and Regards
Abhishek Talwar
|

=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Uns

Re: [openstack-dev] [all] Mitaka Design Summit - Proposed slot allocation

2015-09-04 Thread gord chung




On 04/09/2015 6:14 AM, Thierry Carrez wrote:

Hi PTLs,

Here is the proposed slot allocation for every "big tent" project team
at the Mitaka Design Summit in Tokyo. This is based on the requests the
liberty PTLs have made, space availability and project activity &
collaboration needs.

We have a lot less space (and time slots) in Tokyo compared to
Vancouver, so we were unable to give every team what they wanted. In
particular, there were far more workroom requests than we have
available, so we had to cut down on those quite heavily. Please note
that we'll have a large lunch room with roundtables inside the Design
Summit space that can easily be abused (outside of lunch) as space for
extra discussions.

Here is the allocation:

| fb: fishbowl 40-min slots
| wr: workroom 40-min slots
| cm: Friday contributors meetup
| | day: full day, morn: only morning, aft: only afternoon

Neutron: 12fb, cm:day
Nova: 14fb, cm:day
Cinder: 5fb, 4wr, cm:day
Horizon: 2fb, 7wr, cm:day   
Heat: 4fb, 8wr, cm:morn
Keystone: 7fb, 3wr, cm:day
Ironic: 4fb, 4wr, cm:morn
Oslo: 3fb, 5wr
Rally: 1fb, 2wr
Kolla: 3fb, 5wr, cm:aft
Ceilometer: 2fb, 7wr, cm:morn
TripleO: 2fb, 1wr, cm:full
Sahara: 2fb, 5wr, cm:aft
Murano: 2wr, cm:full
Glance: 3fb, 5wr, cm:full   
Manila: 2fb, 4wr, cm:morn
Magnum: 5fb, 5wr, cm:full   
Swift: 2fb, 12wr, cm:full   
Trove: 2fb, 4wr, cm:aft
Barbican: 2fb, 6wr, cm:aft
Designate: 1fb, 4wr, cm:aft
OpenStackClient: 1fb, 1wr, cm:morn
Mistral: 1fb, 3wr   
Zaqar: 1fb, 3wr
Congress: 3wr
Cue: 1fb, 1wr
Solum: 1fb
Searchlight: 1fb, 1wr
MagnetoDB: won't be present

Infrastructure: 3fb, 4wr (shared meetup with Ironic and QA) 
PuppetOpenStack: 2fb, 3wr
Documentation: 2fb, 4wr, cm:morn
Quality Assurance: 4fb, 4wr, cm:full
OpenStackAnsible: 2fb, 1wr, cm:aft
Release management: 1fb, 1wr (shared meetup with QA)
Security: 2fb, 2wr
ChefOpenstack: will camp in the lunch room all week
App catalog: 1fb, 1wr
I18n: cm:morn
OpenStack UX: 2wr
Packaging-deb: 2wr
Refstack: 2wr
RpmPackaging: 1fb, 1wr

We'll start working on laying out those sessions over the available
rooms and time slots. If you have constraints (I already know
searchlight wants to avoid conflicts with Horizon, Kolla with Magnum,
Manila with Cinder, Solum with Magnum...) please let me know, we'll do
our best to limit them.

this works well for us. thanks! i would say, if possible, please avoid 
overlaps between Ceilometer and Oslo.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

2015-09-03 Thread gord chung




On 28/08/15 01:54 PM, Dan Smith wrote:

we store everything as primitives: floats, time, integer, etc... since
we need to query on attributes. it seems like versionedobjects might not
be useful to our db configuration currently.

I don't think the former determines the latter -- we have lots of things
stored as rows of column primitives and query them out as objects, but
then you're not storing the object and version (unless you do it
separately) So, if it doesn't buy you anything, then there's no reason
to use it.
sorry, i misunderstood this. i thought you were saying ovo may not fit 
into Ceilometer.


i guess to give it more of a real context for us to understand, 
regarding the database layer, if we have an events model which consists of:


- id: uuid
- event_type: string
- generated: timestamp
- raw: dictionary value (not meant for querying, just for auditing purposes)
- traits: [list of tuples (key, value, type)]

given this model, each of our backend drivers take this data and using 
it's connection to db, stores data accordingly:
- in mongodb, the attributes are all stored in documents similar to 
json, raw attr is stored as json
- in elasticsearch, they're stored in documents as well but traits are 
mapped different from mongo

- in hbase, the attributes and traits are all mapped to columns
- in sql, the data is mapped to an Event table, traits are mapped to 
different traits tables depending on type, raw attribute stored as a string.


considering everything is stored differently depending on db, how does 
ovo work? is it normalising it into a specific format pre-storage? how 
does different data will different schemas co-exists on the same db?
- is there a some version tag applied to each item and a version schema 
table created somewhere?
- do we need to migrate the db to some handle different set of 
attributes and what happens for nosql dbs?


also, from api querying pov, if i want to query a db, how do you 
query/filter across different versions?
- does ovo tell the api what versions exists in db and then you can 
filter across any attribute from any schema version?
- are certain attributes effectively unqueryable because it may not 
exists across all versions?


apologies on not understanding how it all works or if the above has 
nothing to do with ovo (i wasn't joking about the 'explain it to me like 
i'm 5' request :-) ) ... i think part of the wariness is that the code 
seemingly does nothing now (or the logic is extremely hidden) but if we 
merge these x hundred/thousand lines of code, it will do something later 
if something changes.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] proposal to add Liusheng to Ceilometer core

2015-09-03 Thread gord chung




On 31/08/15 09:18 AM, gord chung wrote:

hi,

we'd like to nominate Liusheng to the Ceilometer core team. he has 
been a leading contributor in Ceilometer, provides solid reviews, and 
regularly adds ideas for new improvements.


as we did last time, please vote here: 
https://review.openstack.org/#/c/218819/ . if for whatever reason you 
cannot vote there, please respond to this.


reviews:
https://review.openstack.org/#/q/reviewer:liusheng+project:openstack/ceilometer,n,z 



patches:
https://review.openstack.org/#/q/owner:liusheng+status:merged+project:openstack/ceilometer,n,z 



cheers,



it's my pleasure to welcome Liusheng to the Ceilometer core team. thanks 
for the great work you've done and will do.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] proposal to add Pradeep Kilambi to Ceilometer core

2015-09-03 Thread gord chung




On 31/08/15 09:13 AM, gord chung wrote:

hi,

we'd like to nominate Pradeep Kilambi to the Ceilometer core team. he 
has contributed by adding declarative meter support in Ceilometer and 
provides feedback/input in regards to packaging and design.


as we did last time, please vote here: 
https://review.openstack.org/#/c/218822/ . if for whatever reason you 
cannot vote there, please respond to this.


reviews:
https://review.openstack.org/#/q/reviewer:%22Pradeep+Kilambi%22++project:openstack/ceilometer,n,z 



patches:
https://review.openstack.org/#/q/owner:%22Pradeep+Kilambi%22+status:merged+project:openstack/ceilometer,n,z 



cheers,

i'm please to welcome Pradeep to the Ceilometer core team. keep on 
keeping on.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

2015-09-02 Thread gord chung




On 02/09/2015 11:25 AM, Alec Hothan (ahothan) wrote:





On 9/1/15, 11:31 AM, "gord chung"  wrote:


re: serialisation, that probably isn't the biggest concern for
Ceilometer performance. the main items are storage -- to be addressed by
Gnocchi/tsdb, and polling load. i just thought i'd point out an existing
serialisation patch since we were on the topic :-)

Is there any data measuring the polling load on large scale deployments?
Was there a plan to reduce the polling load to an acceptable level? If yes 
could you provide any pointer if any?


i'm not sure any user has provided numbers when raising the issue -- 
just that it's 'high'. this should probably be done in a separate thread 
as i don't want it to get lost in completely unrelated subject. that 
said, an initial patch to minimise load was done in Liberty[1] and 
secondary proposal for M*[2].


conceptually, i would think only the consumers need to know about all 
the queues and even then, it should only really need to know about 
the ones it understands. the producers (polling agents) can just fire 
off to the correct versioned queue and be done... thanks for the 
above link (it'll help with discussion/spec design). 

When everything goes according to plan, any solution can work but this is 
hardly the case in production, especially at scale.  Here are a few question 
that may help in the discussion:
- how are versioned queue named?
- who creates a versioned queue (producer or consumer?) and who deletes it when 
no more entity of that version is running?
- how to make sure a producer is not producing in a queue that has no consumer 
(a messaging infra like rabbit is designed to decouple producers from consumers)
- all corner cases of entities (consumers or producers) popping up with newer 
or older version, and terminating (gracefully or not) during the 
upgrade/downgrade, what happens to the queues...

IMHO using a simple communication schema (1 topic/queue for all versions) with 
in-band message versioning is a much less complex proposition than juggling 
with versioned queues (not to say the former is simple to do). With versioned 
queues you're kind of trading off the per message versioning with per queue 
versioning but at the expense of:
- a complex queue management (if you want to do it right)
- a not less complex per queue message decoding (since the consumer needs to 
know how to decode and interpret every message depending on the version of the 
queue it comes from)
- a more difficult debug environment (harder to debug multiple queues than 1 
queue)
- and added stress on oslo messaging (due to the use of transient queues)

thanks, good items to think about when building spec. will be sure to 
add link when initial draft is ready.


[1] 
https://blueprints.launchpad.net/ceilometer/+spec/resource-metadata-caching

[2] https://review.openstack.org/#/c/209799/

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

2015-09-01 Thread gord chung




On 28/08/2015 5:18 PM, Alec Hothan (ahothan) wrote:





On 8/28/15, 11:39 AM, "gord chung"  wrote:


i should start by saying i re-read my subject line and it arguably comes
off aggressive -- i should probably have dropped 'explain' :)

On 28/08/15 01:47 PM, Alec Hothan (ahothan) wrote:

On 8/28/15, 10:07 AM, "gord chung"  wrote:


On 28/08/15 12:18 PM, Roman Dobosz wrote:

So imagine we have new versions of the schema for the events, alarms or
samples in ceilometer introduced in Mitaka release while you have all
your ceilo services on Liberty release. To upgrade ceilometer you'll
have to stop all services to avoid data corruption. With
versionedobjects you can do this one by one without disrupting
telemetry jobs.

are versions checked for every single message? has anyone considered the
overhead to validating each message? since ceilometer is queue based, we
could technically just publish to a new queue when schema changes... and
the consuming services will listen to the queue it knows of.

ie. our notification service changes schema so it will now publish to a
v2 queue, the existing collector service consumes the v1 queue until
done at which point you can upgrade it and it will listen to v2 queue.

this way there is no need to validate/convert anything and you can still
take services down one at a time. this support doesn't exist currently
(i just randomly thought of it) but assuming there's no flaw in my idea
(which there may be) isn't this more efficient?

If high performance is a concern for ceilometer (and it should) then maybe
there might be better options than JSON?
JSON is great for many applications but can be inappropriate for other
demanding applications.
There are other popular open source encoding options that yield much more
compact wire payload, more efficient encoding/decoding and handle
versioning to a reasonable extent.

i should clarify. we let oslo.messaging serialise our dictionary how it
does... i believe it's JSON. i'd be interested to switch it to something
more efficient. maybe it's time we revive the msgpacks patch[1] or are
there better alternatives? (hoping i didn't just unleash a storm of
'this is better' replies)

I'd be curious to know if there is any benchmark on the oslo serializer for 
msgpack and how it compares to JSON?
More important is to make sure we're optimizing in the right area.
Do we have a good understanding of where ceilometer needs to improve to scale 
or is it still not quite clear cut?


re: serialisation, that probably isn't the biggest concern for 
Ceilometer performance. the main items are storage -- to be addressed by 
Gnocchi/tsdb, and polling load. i just thought i'd point out an existing 
serialisation patch since we were on the topic :-)





Queue based versioning might be less runtime overhead per message but at
the expense of a potentially complex queue version management (which can
become tricky if you have more than 2 versions).
I think Neutron was considering to use versioned queues as well for its
rolling upgrade (along with versioned objects) and I already pointed out
that managing the queues could be tricky.

In general, trying to provide a versioning framework that allows to do
arbitrary changes between versions is quite difficult (and often bound to
fail).


yeah, so that's what a lot of the devs are debating about right now.
performance is our key driver so if we do something we think/know will
negatively impact performance, it better bring a whole lot more of
something else. if queue based versioning offers comparable
functionalities, i'd personally be more interested to explore that route
first. is there a thread/patch/log that we could read to see what
Neutron discovered when they looked into it?

The versioning comments are buried in this mega patch if you are brave enough 
to dig in:

https://review.openstack.org/#/c/190635

The (offline) conclusion was that this was WIP and deserved more discussion 
(need to check back with Miguel and Ihar from the Neutron team).
One option considered in that discussion was to use oslo messaging topics to 
manage flows of messages that had different versions (and still use 
versionedobjects). So if you have 3 versions in your cloud you'd end up with 3 
topics (and as many queues when it comes to Rabbit). What is complex is to 
manage the queues/topic names (how to name them), how to discover them and how 
to deal with all the corner cases (like a new node coming in with an arbitrary 
version, nodes going away at any moment, downgrade cases).


conceptually, i would think only the consumers need to know about all 
the queues and even then, it should only really need to know about the 
ones it understands. the producers (polling agents) can just fire off to 
the correct versioned queue and be done... thanks for the above link 
(it'll help with discussion/spec design).


cheers,

Re: [openstack-dev] [nova][ceilometer] cputime value resets on restart/shutdown

2015-09-01 Thread gord chung




On 01/09/2015 12:10 PM, Julien Danjou wrote:

On Tue, Sep 01 2015, gord chung wrote:


but if it's not actually cumulative in Ceilometer (pre-storage), should we
really be tagging it as such?

We only have 3 meters type, and the cumulative definition I wrote
somewhere back in 2012 states that it can reset to 0. Sorry. :-)


we shouldn't redefine real words -- no one reads definitions for words 
they already know... also, i'm not sure the 'reset to 0' definition is 
easily accessible (i'm not sure where it exists) :-\





so i was thinking rather than modify the existing meter, we build a new
cpu.delta meter, which is gives the delta. it seems like giving a delta meter
would make the behaviour more consistent.

…with data loss if you restart the polling agent and it then ignores the
previous value of the meter.

Except if you connect the polling system to the previous state of the
meter, which requires to either:
1. Connect the polling system to the storage system
2. Store locally the latest value you fetched

Option 1 sounds crazy, option 2 sounds less crazy, but still hackish.

Whereas having a system that can compute the delta afterward with all
the value at its reach sounds way better – that's why I'm in favor of
doing that in the storage system (e.g. Gnocchi).

i realise now we're just rehashing the same topic from 3 years ago[1]. 
it seemed like the issue then was we couldn't scale out multiple 
notification consumers... which we can now.


i agree that (1) is bad. (2) has potential data loss on agent restart... 
that said, you already have data loss regardless as when you poll 
'cumulative' meter, the cputime between poll cycles is loss during 
restart. if you handle this at storage level, every subsequent sample 
will be be off by an arbitrary amount after each restart where the delta 
solution, you'll only be off a single sample whenever instance/agent 
restarts.


[1]http://lists.openstack.org/pipermail/openstack-dev/2012-November/003297.html

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer] cputime value resets on restart/shutdown

2015-09-01 Thread gord chung




On 31/08/2015 3:36 PM, Julien Danjou wrote:

On Mon, Aug 31 2015, gord chung wrote:


i'm not sure Gnocchi is where we should be fixing this as it really only
(potentially) fixes it for Gnocchi and not for any of the other ways Ceilometer
data can be consumed.

The ideal way is to send the data as it is collected and do the
treatment in the backend, otherwise you end up implementing the world in
Ceilometer.


but if it's not actually cumulative in Ceilometer (pre-storage), should 
we really be tagging it as such?



a proposed solution in bug and in a previous thread suggests that a 'delta'
meter be built from cputime value libvirt provides which would better handle
the reset scenario. that said, is there another option to truly have a
cumulative meter?

Yes, the hackish way might be to convert those cumulative to delta with
a transfomer in the pipeline and be done with it. Though you'll probably
have data points missing on restart, which is not ultimate solution. As
soon as you start losing information in the Ceilometer pipeline by
transforming I'll stand and say "bad idea". :-)


so i was thinking rather than modify the existing meter, we build a new 
cpu.delta meter, which is gives the delta. it seems like giving a delta 
meter would make the behaviour more consistent.




So yes, you have to have a smart storage solution in the end.

ideally, i was thinking Nova would capture this data and there would be 
absolutely no data loss as Nova knows exactly when instances are 
restarted/stopped... that said, it's arguably outside the scope of Nova.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer] cputime value resets on restart/shutdown

2015-08-31 Thread gord chung




On 31/08/2015 1:06 PM, Julien Danjou wrote:

On Mon, Aug 31 2015, gord chung wrote:


for context, we have an open bug in Ceilometer where the value used to capture
cputime of an instance is being reset whenever the instance is restarted or
shutdown[1]. this seems to be the expected behaviour for libvirt driver[2] but
it conflicts with the fact that the meter is currently defined as 'cumulative'
-- which by definition, should not reset. it seems the correct meter type is a
'gauge'.

Nah, gauge is for things with discrete values, e.g. a number of objects
in Swift.
Cumulative are counters that increase and that can be reset.
The problem is that the API still don't know to detect resets and
normalize the values.

That's what needs fixing. We could fix it with Gnocchi.

i think that's the reason the bug is raised... that's not what 
cumulative means. by definition it should increase over each successive 
value. it's cumulative from the pov of qemu process but not from 
instance pov. judging by how Hyper-V meter is built, it doesn't seem 
like Hyper-V's cputime meter would ever reset.


i'm not sure Gnocchi is where we should be fixing this as it really only 
(potentially) fixes it for Gnocchi and not for any of the other ways 
Ceilometer data can be consumed.


a proposed solution in bug and in a previous thread suggests that a 
'delta' meter be built from cputime value libvirt provides which would 
better handle the reset scenario. that said, is there another option to 
truly have a cumulative meter?


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][ceilometer] cputime value resets on restart/shutdown

2015-08-31 Thread gord chung


hi

i'm reaching out for help understanding meter values across hypervisors.

for context, we have an open bug in Ceilometer where the value used to 
capture cputime of an instance is being reset whenever the instance is 
restarted or shutdown[1]. this seems to be the expected behaviour for 
libvirt driver[2] but it conflicts with the fact that the meter is 
currently defined as 'cumulative' -- which by definition, should not 
reset. it seems the correct meter type is a 'gauge'.


with that said, for other hypervisors: vmware, Xen, hyper-v, etc... does 
cputime also reset when an instance restarts/shuts down? is there a way 
to find a truly cumulative value for cputime for libvirt and other 
hypervisors that reset?


[1] https://bugs.launchpad.net/ceilometer/+bug/1417949
[2] https://www.redhat.com/archives/libvirt-users/2012-April/msg00100.html

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] proposal to add Liusheng to Ceilometer core

2015-08-31 Thread gord chung


hi,

we'd like to nominate Liusheng to the Ceilometer core team. he has been 
a leading contributor in Ceilometer, provides solid reviews, and 
regularly adds ideas for new improvements.


as we did last time, please vote here: 
https://review.openstack.org/#/c/218819/ . if for whatever reason you 
cannot vote there, please respond to this.


reviews:
https://review.openstack.org/#/q/reviewer:liusheng+project:openstack/ceilometer,n,z

patches:
https://review.openstack.org/#/q/owner:liusheng+status:merged+project:openstack/ceilometer,n,z

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] proposal to add Pradeep Kilambi to Ceilometer core

2015-08-31 Thread gord chung


hi,

we'd like to nominate Pradeep Kilambi to the Ceilometer core team. he 
has contributed by adding declarative meter support in Ceilometer and 
provides feedback/input in regards to packaging and design.


as we did last time, please vote here: 
https://review.openstack.org/#/c/218822/ . if for whatever reason you 
cannot vote there, please respond to this.


reviews:
https://review.openstack.org/#/q/reviewer:%22Pradeep+Kilambi%22++project:openstack/ceilometer,n,z

patches:
https://review.openstack.org/#/q/owner:%22Pradeep+Kilambi%22+status:merged+project:openstack/ceilometer,n,z

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Gnocchi] Added Ilya Tyaptin as core reviewer

2015-08-28 Thread gord chung



On 28/08/15 02:48 PM, Julien Danjou wrote:

Hi fellows,

Ilya did a few good contributions to Gnocchi, especially around the
InfluxDB driver, so I'm glad to add him to the list of core reviewers.

Welcome aboard.

Cheers,


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

well deserved Ilya!

--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

2015-08-28 Thread gord chung

i should start by saying i re-read my subject line and it arguably comes 
off aggressive -- i should probably have dropped 'explain' :)


On 28/08/15 01:47 PM, Alec Hothan (ahothan) wrote:


On 8/28/15, 10:07 AM, "gord chung"  wrote:



On 28/08/15 12:18 PM, Roman Dobosz wrote:

So imagine we have new versions of the schema for the events, alarms or
samples in ceilometer introduced in Mitaka release while you have all
your ceilo services on Liberty release. To upgrade ceilometer you'll
have to stop all services to avoid data corruption. With
versionedobjects you can do this one by one without disrupting
telemetry jobs.

are versions checked for every single message? has anyone considered the
overhead to validating each message? since ceilometer is queue based, we
could technically just publish to a new queue when schema changes... and
the consuming services will listen to the queue it knows of.

ie. our notification service changes schema so it will now publish to a
v2 queue, the existing collector service consumes the v1 queue until
done at which point you can upgrade it and it will listen to v2 queue.

this way there is no need to validate/convert anything and you can still
take services down one at a time. this support doesn't exist currently
(i just randomly thought of it) but assuming there's no flaw in my idea
(which there may be) isn't this more efficient?

If high performance is a concern for ceilometer (and it should) then maybe
there might be better options than JSON?
JSON is great for many applications but can be inappropriate for other
demanding applications.
There are other popular open source encoding options that yield much more
compact wire payload, more efficient encoding/decoding and handle
versioning to a reasonable extent.


i should clarify. we let oslo.messaging serialise our dictionary how it 
does... i believe it's JSON. i'd be interested to switch it to something 
more efficient. maybe it's time we revive the msgpacks patch[1] or are 
there better alternatives? (hoping i didn't just unleash a storm of 
'this is better' replies)




Queue based versioning might be less runtime overhead per message but at
the expense of a potentially complex queue version management (which can
become tricky if you have more than 2 versions).
I think Neutron was considering to use versioned queues as well for its
rolling upgrade (along with versioned objects) and I already pointed out
that managing the queues could be tricky.

In general, trying to provide a versioning framework that allows to do
arbitrary changes between versions is quite difficult (and often bound to
fail).

yeah, so that's what a lot of the devs are debating about right now. 
performance is our key driver so if we do something we think/know will 
negatively impact performance, it better bring a whole lot more of 
something else. if queue based versioning offers comparable 
functionalities, i'd personally be more interested to explore that route 
first. is there a thread/patch/log that we could read to see what 
Neutron discovered when they looked into it?


[1] https://review.openstack.org/#/c/151301/

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

2015-08-28 Thread gord chung




On 28/08/15 12:49 PM, Dan Smith wrote:

there was a little skeptism because it was originally sold as magic,
but reading the slides from Vancouver[1], it is not magic.

I think I specifically said "they're not magic" in my slides. Not sure
who sold you them as magic, but you should leave them a
less-than-five-stars review.


i like how your slides leveled it for us :)



Ceilometer functions mainly on queue-based IPC. most of the
communication is async transferring of json payloads where callback is
not required. the basic workflows are:

This is specifically something versionedobjects should help with. The
remotable RPC method calls on an object are something that nova uses
heavily, but other projects don't use at all.


polling agent --- topic queue ---> notification agent --- topic queue
---> collector (direct connection to db)

What happens if any of these components are running different versions
of the ceilometer code at one point? During an upgrade, you presumably
don't want to have to take all of these things down at once, and so the
"notification agent" might get an object from the "polling agent" that
is older or newer than it expects. More specifically, maybe the
collector is writing to older schema and gets a newer object from the
front of the queue with data it can't store. If you're getting
versionedobjects instead of raw json, you at least have an indication
that this is happening. If you get an older object, you might choose to
do something specific for the fields that are now in the DB schema, but
aren't in the object you received.


OpenStack service --- topic queue ---> notification agent --- topic
queue ---> collector (direct connection to db)

This is a good one. If Nova was sending notifications as objects, then
the notification agent would get a version with each notification,
knowing specifically when the notification is newer than it supports,
instead of us just changing things (on purpose or by accident) and you
breaking.

 From the storage in the DB perspective, I'm not sure what your
persistence looks like. However, we've been storing _some_ things in our
DB as serialized objects. That means that if we pull something out in a
year, after which time things in the actual object implementation have
changed, then we have an indication of what version it was stored in,
and presumably can apply a process to update it (or handle the
differences) at load time. I'm not sure if that's useful for ceilometer,
but it is definitely useful for nova, where we can avoid converting
everything in the database every time we add/change a field in something
-- a process that is very critical to avoid in our goals for improving
the upgrade experience for operators.


we store everything as primitives: floats, time, integer, etc... since 
we need to query on attributes. it seems like versionedobjects might not 
be useful to our db configuration currently.




So, I dunno if ceilometer needs to adopt versionedobjects for anything.
It seems like it would apply to the cases you describe above, but if
not, there's no reason to use it just because others are. Nova will
(when I stop putting it off) start sending notifications as serialized
and versioned objects at some point, but you may choose to just unwrap
it and treat it as a json blob beyond the handler, if that's what is
determined as the best course.


i'm really looking forward to this. i think the entire Ceilometer team 
is waiting for someone to contractualise messages. right now it's a crap 
shoot when we listen to messages from other services.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

2015-08-28 Thread gord chung




On 28/08/15 12:18 PM, Roman Dobosz wrote:

So imagine we have new versions of the schema for the events, alarms or
samples in ceilometer introduced in Mitaka release while you have all
your ceilo services on Liberty release. To upgrade ceilometer you'll
have to stop all services to avoid data corruption. With
versionedobjects you can do this one by one without disrupting
telemetry jobs.
are versions checked for every single message? has anyone considered the 
overhead to validating each message? since ceilometer is queue based, we 
could technically just publish to a new queue when schema changes... and 
the consuming services will listen to the queue it knows of.


ie. our notification service changes schema so it will now publish to a 
v2 queue, the existing collector service consumes the v1 queue until 
done at which point you can upgrade it and it will listen to v2 queue.


this way there is no need to validate/convert anything and you can still 
take services down one at a time. this support doesn't exist currently 
(i just randomly thought of it) but assuming there's no flaw in my idea 
(which there may be) isn't this more efficient?


The other thing, maybe not so obvious, is to put versionedobject layer
between application and the MongoDB driver, so that all of the schema
changes will be automatically handled on ovo, and also serialization
might also be done on such layer.


i don't quite understand this, is this a mongodb specific solution? 
admittedly the schemaless design of mongo i can imagine causing issues 
but currently we're trying to avoid wasting resources on existing 
mongodb solution as we attempt to move to new api. if it's just a 
generic db solution, i'd be interested to apply it to future designs.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

2015-08-27 Thread gord chung


hi,

there has been a lot of work done across the community and Ceilometer 
relating to versionedobjects. in Ceilometer particularly, this effort 
has somewhat stalled as contributors are unsure of the benefits of 
versionedobjects and how it relates to Ceilometer. there was a little 
skeptism because it was originally sold as magic, but reading the slides 
from Vancouver[1], it is not magic. rather it seems the main purpose is 
to handle the evolution of schemas specifically over RPC which seems 
neat but conceptually doesn't seem to fit into how Ceilometer functions.


looking at the patches, Chris brought up a good question in a review[2] 
which to summarise:


If the ceilometer/aodh tools have direct connections to their data 
storage level (they do) and do not use storable distributed objects 
(on which rpc calls are made) in what sense are versioned objects 
useful to the service?


My understanding is that in Nova (for example) versioned objects are 
useful because rpc calls are made on storable objects that can be in 
flight at any time across the distributed service and thus for their 
to be smooth rolling upgrades those in flight objects need to be able 
to be of different versions.


Ceilometer functions mainly on queue-based IPC. most of the 
communication is async transferring of json payloads where callback is 
not required. the basic workflows are:


polling agent --- topic queue ---> notification agent --- topic queue 
---> collector (direct connection to db)

or
OpenStack service --- topic queue ---> notification agent --- topic 
queue ---> collector (direct connection to db)

or
from Aodh/alarming pov:
ceilometer-api (direct connection to db) <--- http --- alarm evaluator 
--- rpc ---> alarm notifier --- http ---> [Heat/other]


based on the above workflows, is there a good place for adoption of 
versionedobjects? and if so, what is the benefit? most of us are keen on 
adopting consistent design practices but none of us can honestly 
determine why versionedobjects would be beneficial to Ceilometer. if 
someone could explain it to us like we are 5 -- it's probably best to 
explain everything/anything like i'm 5 -- that would help immensely on 
moving this work forward.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cross-project meeting times

2015-08-21 Thread gord chung




On 21/08/2015 4:58 AM, Thierry Carrez wrote:

Anne Gentle wrote:

Hi all,

In last week's TC Highlights blog post [1] I asked if there is interest
in moving the cross-project meeting. Historically it is held after the
TC meeting, but there isn't a requirement for those timings to line up.
I've heard from European and Eastern Standard Time contributors that
it's a tough time to meet half the year. It's also a bit early for APAC,
my apologies for noting this but still proposing to meet earlier.

I'd like to propose a new cross-project meeting time, 1800 Tuesdays. To
that end I've created a review with the proposed time:

https://review.openstack.org/214605

Please take a look, see if you think it could work, and let us know
either on this list or the review itself.

Commented on the review... I think 1800 UTC is not significantly more
convenient for Europeans (dinner hours between 1700 and 1900 UTC)
compared to 2100 UTC. It makes it more convenient for East-of-Moscow
Russians, but we lose Australia in the process.

If we are to lose Australia anyway, I would move even earlier (say 15:00
or 16:00 UTC) and cover China <-> US West. That could be a good rotation
with the one at 21:00 UTC which covers Australia <-> West Europe.



+1 for even earlier. it requires west coast Americas to wake up early 
but the biggest time gap is the Pacific. that said, it's entirely 
possible most of those participating are from west coast...


maybe we should shift more emphasis to ML? working with non-native 
English speaking companies, i know they are very interested in 
participating but considering the oft hectic pace of the 'live' 
meetings, they tend to be viewers as they can't get their ideas into 
English fast enough.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer][AODH] Timeout Event Alarms

2015-08-17 Thread gord chung


good questions...

On 17/08/2015 10:19 AM, Igor Degtiarov wrote:

Gordon probably that question to you:
Are we going to create a new folder in spec's dir for next cycle, or 
we continue discussing "new" specs as part of liberty?


we can create a new dir for M* cycle specs. as mentioned in last week's 
meeting, even though we don't have a exact date for feature freeze, 
unless it's a small bug size spec, Liberty specs are closed. feel free 
to create a patch for new M* dir


And the second one: are we going to create special rep for AODH specs 
or ceilometer-specs is ok for all new projects?


we'll track aodh specs under ceilometer now -- i don't want to increase 
the number of repos we need to monitor unless we notice Aodh specs are 
too much to handle.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] deprecating RPC publisher/collector

2015-08-11 Thread gord chung


hi,

this note is to raise awareness that the rpc publishing functionality in 
Ceilometer will be deprecated and disabled in Liberty[1] as it offers 
redundant capabilities vs notifier publishing.


back in Juno, we adopted oslo.messaging and switched from RPC to work 
queues in Ceilometer -- the notifier (queues) publisher was made the 
default means of communication between Ceilometer services. the benefits 
of RPC is that it offers callback functionality but it is often at the 
expense of performance [2]. As Ceilometer doesn't need the callback 
functionality, it was advised to switch from rpc to notifier publisher.


regarding support going forward, after some discussion, it's our hope 
that since we switched out rpc so long ago, the rpc publisher will be 
removed in M* cycle. if there are any concerns, please raise them here. :)


[1] https://review.openstack.org/#/c/211304/
[2] https://www.rabbitmq.com/tutorials/tutorial-six-python.html

cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] [aodh] upgrade path

2015-08-07 Thread gord chung




On 07/08/2015 3:49 AM, Chris Dent wrote:


Despite our conversation in the meeting yesterday[1] I still remain a
bit confused about the upgrade path from alarming-in-ceilometer to
alarming provided by aodh and the availability of the older code in
released liberty.

Much of my confusion can probably be resolved by knowing the answer to
this question:

If someone installs aodh on a machine that already has ceilometer on it
and turns off ceilometer-alarm-notifier and ceilometer-alarm-evaluator
(in favor of aodh-notifier and aodh-evaluator) will they be able to run
those aodh services against their existing ceilometer.conf files[2]?

What if they were, in ceilometer, using specific config for their
alarming database (alarm_connection). Can aodh "see" and use this
config option?

Or will they need to copy and modify the existing conf files to allow
them to carry on using their existing database config?

I know that I can go try some of this in a devstack, but that's not
really the point of the question. The point is: What are we expecting
existing deployments to do?

I had assumed that the reason for keeping alarming in ceilometer for
the liberty release was to allow a deployment to upgrade to Liberty
across the project suites without having to go through modifying
alarming config and managing an alarming migration in the same step.
That migration ought to be pretty minor (tweak a config here and
there) but unless the answer to my first question is "yes" it has some
significance.


following up, after speaking with Chris, a critical question was not 
just what happens to those who upgrade but what happens to those who 
choose NOT to upgrade to Aodh. to clarify, it is Ceilometer's intent to 
have Aodh as the source of alarming functionality going forward -- no 
new features have been added or will be added to the existing alarming 
code in Ceilometer. also, any new feature must be added to Aodh.


with that said, for those who choose not to upgrade and are content with 
existing alarming code, the code will exist as is for Liberty. after 
speaking with the Nova team, there has been a deprecation period between 
Cinder/Glance split before it was fully removed from packaging/code. 
Ceilometer will follow the same but will target a more aggressive 
deprecation period and the code will be removed in M* cycle.


the code removal is dependent on Aodh being gated on, released and 
packaged. it is also dependent on any upgrade requirements being documented.


the goals for a short deprecation is:
- to avoid a slow complicated divergence in code that will lead to 
difficult maintenance

- to allow time for packagers to package the new Aodh service
- to give operators, tracking the latest and greatest, the option of 
whether to upgrade to Aodh or not.


i hope that clarifies our intentions. this is our first split so if 
there are any noticeable gaps in logic, please feel free to chime in.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] [aodh] upgrade path

2015-08-07 Thread gord chung




On 07/08/2015 3:49 AM, Chris Dent wrote:


Despite our conversation in the meeting yesterday[1] I still remain a
bit confused about the upgrade path from alarming-in-ceilometer to
alarming provided by aodh and the availability of the older code in
released liberty.

Much of my confusion can probably be resolved by knowing the answer to
this question:

If someone installs aodh on a machine that already has ceilometer on it
and turns off ceilometer-alarm-notifier and ceilometer-alarm-evaluator
(in favor of aodh-notifier and aodh-evaluator) will they be able to run
those aodh services against their existing ceilometer.conf files[2]?


it also depends on how you're consuming ceilometer. if you're installing 
ceilometer services via packages, there will never be aodh or 
ceilometer-alarm... just one alarming service. once we get aodh 
released/packaged, from a package pov, the current, deprecated code will 
be inaccessible.


http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/%23openstack-meeting.2015-08-06.log.html#t2015-08-06T15:46:17 



--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer][AODH] Timeout Event Alarms

2015-08-04 Thread gord chung


hi Igor,

i would suggest you go with second option as i believe your 
implementation will overlap and reuse some of the functionality Ryota 
would code for his alarm spec [1]. also, since Aodh is working on an 
independent release cycle, it'll give you some more time as i don't 
think we'd be able to get this into Liberty if we went the pipeline route.


[1] 
http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/event-alarm-evaluator.html


On 04/08/2015 10:00 AM, Igor Degtiarov wrote:

Hi folks,

On our meatup we agreed to add timeout event alarms [1](Event-Base 
Alarming part).

In ToDo task "Сhoose the optimal way for timeout alerting implementation"
Now we have two proposition for implementation:
 - first is to add timeout param in event pipeline (transformer part) [2]
   -- weakness of this approach is that we cannot allow user change 
config files, so only administrator will be able to set rules for 
timeout events alarms, and that is not what we are expecting from alarms.
 - second is additional optional parameters in event alarms 
description like sequence of required events and timeout threshold. 
Event alarm evaluator looks thru getting events and evaluates alarm if 
even one event from required sequence isn't received in set "timeout".[3]


It seems that second approach is better it doesn't have restrictions 
for end user.

Hope for your help in choosing optimal way for implementation.
(In specs review there is silence now)

[1] 
https://wiki.openstack.org/wiki/Meetings/Ceilometer/Liberty_Virtual_Mid-Cycle

[2] https://review.openstack.org/#/c/162167
[3] https://review.openstack.org/#/c/199005

Igor Degtiarov
Software Engineer
Mirantis Inc.
www.mirantis.com 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

1 2 >

1 - 100 of 105 matches

Mail list logo