[openstack-dev] [Ceilometer] Capturing isolated events

[Sachin Manpathak]

Can ceilometer-gurus advise?

[For Ceilometer/Juno]
Use case:
catch instance creation error notifications with ceilometer.

I tried the notifications as well as metering route, but haven't quite
figured out how to do this --

- Added a notification class to ceilometer
/
compute

/notifications

/instance.py

- Noticed that there is a class called "Instance" which already subscribes
to events compute.instance.*, disabled that by removing it from the egg
entry points (setup.cfg)
- Created a pipeline to define a meter for my notifications class.

I noticed that the ceilometer-agent-notification service does not consume
any notifications with above changes. I had to set
store_events=true in ceilometer.conf

Now, ceilometer generates events, but they are generic type.

1. Should I change event_definitions.yaml to store event information in the
format that I want?
2. a way to generate only the event I am interested in, not all by default.


Thanks,
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] In memory joins in Nova

[Sachin Manpathak]

Thanks, This feedback was helpful.
Perhaps my paraphrasing was misleading. I am not running openstack at scale
in order to see how much the DB can sustain. My observation was that the
host running nova services saturates on CPU much earlier than the DB does.
Joins could be one of the reasons. I also observed that background tasks
like instance creation, resource/stats updates contend with get queries. In
addition to caching optimizations prioritizing tasks in nova could help.

Is there a nova API to fetch list of instances without metadata? Until I
find a good way to profile openstack code, changing the queries can be a
good experiement IMO.



On Wed, Aug 12, 2015 at 8:12 AM, Dan Smith  wrote:

> > If OTOH we are referring to the width of the columns and the join is
> > such that you're going to get the same A identity over and over again,
> > if you join A and B you get a "wide" row with all of A and B with a very
> > large amount of redundant data sent over the wire again and again (note
> > that the database drivers available to us in Python always send all rows
> > and columns over the wire unconditionally, whether or not we fetch them
> > in application code).
>
> Yep, it was this. N instances times M rows of metadata each. If you pull
> 100 instances and they each have 30 rows of system metadata, that's a
> lot of data, and most of it is the instance being repeated 30 times for
> each metadata row. When we first released code doing this, a prominent
> host immediately raised the red flag because their DB traffic shot
> through the roof.
>
> > In this case you *do* want to do the join in
> > Python to some extent, though you use the database to deliver the
> > simplest information possible to work with first; you get the full row
> > for all of the A entries, then a second query for all of B plus A's
> > primary key that can be quickly matched to that of A.
>
> This is what we're doing. Fetch the list of instances that match the
> filters, then for the ones that were returned, get their metadata.
>
> --Dan
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] In memory joins in Nova

[Sachin Manpathak]

I am struggling with python code profiling in general. It has its own
caveats like 100% plus overhead.
However, on a host with only nova services (DB on a different host), I see
cpu utilization spike up quickly with scale. The DB server is relatively
calm and never goes over 20%. On a system which relies on DB to fetch all
the data, this should not happen.

I could not find any analysis of nova performance either. Appreciate if
someone can point me to one.

Thanks,





On Tue, Aug 11, 2015 at 3:57 PM, Chris Friesen 
wrote:

> Just curious...have you measured this consuming a significant amount of
> CPU time?  Or is it more a gut feel of "this looks like it might be
> expensive"?
>
> Chris
>
>
> On 08/11/2015 04:51 PM, Sachin Manpathak wrote:
>
>> Here are a few --
>> instance_get_all_by_filters joins manually with
>> instances_fill_metadata --
>>
>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1890
>>
>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1782
>>
>> Almost all instance query functions manually join with instance_metadata.
>>
>> Another example was compute_node_get_all function which joined
>> compute_node,
>> services and ip tables. But it is simplified  in current codebase (I am
>> working
>> on Juno)
>>
>>
>>
>>
>> On Tue, Aug 11, 2015 at 3:09 PM, Clint Byrum > <mailto:cl...@fewbar.com>> wrote:
>>
>> Excerpts from Sachin Manpathak's message of 2015-08-12 05:40:36 +0800:
>> > Hi folks,
>> > Nova codebase seems to follow manual joins model where all data
>> required by
>> > an API is fetched from multiple tables and then joined manually by
>> using
>> > (in most cases) python dictionary lookups.
>> >
>> > I was wondering about the basis reasoning for doing so. I usually
>> find
>> > openstack services to be CPU bound in a medium sized environment and
>> > non-trivial utilization seems to be from parts of code which do
>> manual
>> > joins.
>>
>> Could you please cite specific examples so we can follow along with
>> your
>> thinking without having to repeat your analysis?
>>
>> Thanks!
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] In memory joins in Nova

[Sachin Manpathak]

Here are a few --
instance_get_all_by_filters joins manually with
instances_fill_metadata --
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1890
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1782

Almost all instance query functions manually join with instance_metadata.

Another example was compute_node_get_all function which joined
compute_node, services and ip tables. But it is simplified  in current
codebase (I am working on Juno)




On Tue, Aug 11, 2015 at 3:09 PM, Clint Byrum  wrote:

> Excerpts from Sachin Manpathak's message of 2015-08-12 05:40:36 +0800:
> > Hi folks,
> > Nova codebase seems to follow manual joins model where all data required
> by
> > an API is fetched from multiple tables and then joined manually by using
> > (in most cases) python dictionary lookups.
> >
> > I was wondering about the basis reasoning for doing so. I usually find
> > openstack services to be CPU bound in a medium sized environment and
> > non-trivial utilization seems to be from parts of code which do manual
> > joins.
>
> Could you please cite specific examples so we can follow along with your
> thinking without having to repeat your analysis?
>
> Thanks!
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] In memory joins in Nova

[Sachin Manpathak]

Hi folks,
Nova codebase seems to follow manual joins model where all data required by
an API is fetched from multiple tables and then joined manually by using
(in most cases) python dictionary lookups.

I was wondering about the basis reasoning for doing so. I usually find
openstack services to be CPU bound in a medium sized environment and
non-trivial utilization seems to be from parts of code which do manual
joins.

Thanks,
-Sachin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev