Re: [openstack-dev] Event Service

2013-07-14 Thread Sandy Walsh
https://wiki.openstack.org/wiki/SystemUsageData Both Ceilometer and StackTach can be used to consume these notifications. https://github.com/openstack/ceilometer https://github.com/rackerlabs/stacktach (StackTach functionality is slowly being merged into Ceilometer) Hope it helps! -S

Re: [openstack-dev] Cheetah vs Jinja

2013-07-16 Thread Sandy Walsh
I've used jinja2 on many projects ... it's always been solid. -S From: Solly Ross [sr...@redhat.com] Sent: Tuesday, July 16, 2013 10:41 AM To: OpenStack Development Mailing List Subject: [openstack-dev] [Change I30b127d6] Cheetah vs Jinja (This email is w

Re: [openstack-dev] [Change I30b127d6] Cheetah vs Jinja

2013-07-16 Thread Sandy Walsh
There's a ton of reviews/comparisons out there, only a google away. From: Doug Hellmann [doug.hellm...@dreamhost.com] Sent: Tuesday, July 16, 2013 1:45 PM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [Change I30b127d6] Cheetah vs Jinja Grea

[openstack-dev] Opinions needed: Changing method signature in RPC callback ...

2013-07-18 Thread Sandy Walsh
Hey y'all! Running into an interesting little dilemma with a branch I'm working on. Recently, I introduced a branch in oslo-common to optionally .reject() a kombu message on an exception. Currently, we always .ack() all messages even if the processing callback fails. For Ceilometer, this is a prob

Re: [openstack-dev] Opinions needed: Changing method signature in RPC callback ...

2013-07-18 Thread Sandy Walsh
On 07/18/2013 11:09 AM, Sandy Walsh wrote: > 2. make a generic CallContext() object to include with message that has > anything else we need (a one-time signature break) > > call_context = CallContext({"delivery_info": {...}, "wait": False}) > callback(messa

Re: [openstack-dev] Opinions needed: Changing method signature in RPC callback ...

2013-07-18 Thread Sandy Walsh
On 07/18/2013 12:26 PM, Kevin L. Mitchell wrote: > On Thu, 2013-07-18 at 11:09 -0300, Sandy Walsh wrote: >> 3. some other ugly python hack that I haven't thought of yet. > > I have two possibilities, though neither is really ideal. One is to > provide a duplicate c

Re: [openstack-dev] Opinions needed: Changing method signature in RPC callback ...

2013-07-18 Thread Sandy Walsh
On 07/18/2013 03:55 PM, Eric Windisch wrote: > On Thu, Jul 18, 2013 at 10:09 AM, Sandy Walsh <mailto:sandy.wa...@rackspace.com>> wrote: > > > My worry is busting all the other callbacks out there that use > olso-common.rpc > > > These callback met

Re: [openstack-dev] Opinions needed: Changing method signature in RPC callback ...

2013-07-19 Thread Sandy Walsh
On 07/18/2013 05:56 PM, Eric Windisch wrote: > > > These callback methods are part of the Kombu driver (and maybe part of > > Qpid), but are NOT part of the RPC abstraction. These are private > > methods. They can be broken for external consumers of these methods, > > because the

Re: [openstack-dev] [Nova] New DB column or new DB table?

2013-07-19 Thread Sandy Walsh
On 07/18/2013 11:12 PM, Lu, Lianhao wrote: > Sean Dague wrote on 2013-07-18: >> On 07/17/2013 10:54 PM, Lu, Lianhao wrote: >>> Hi fellows, >>> >>> Currently we're implementing the BP >>> https://blueprints.launchpad.net/nova/+spec/utilization-aware-scheduling. >>> The main idea is to have >> an

Re: [openstack-dev] [Nova] New DB column or new DB table?

2013-07-19 Thread Sandy Walsh
On 07/19/2013 09:43 AM, Sandy Walsh wrote: > > > On 07/18/2013 11:12 PM, Lu, Lianhao wrote: >> Sean Dague wrote on 2013-07-18: >>> On 07/17/2013 10:54 PM, Lu, Lianhao wrote: >>>> Hi fellows, >>>> >>>> Currently we're impl

Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling

2013-07-19 Thread Sandy Walsh
On 07/19/2013 09:47 AM, Day, Phil wrote: >> -Original Message- >> From: Sean Dague [mailto:s...@dague.net] >> Sent: 19 July 2013 12:04 >> To: OpenStack Development Mailing List >> Subject: Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics >> collector for scheduling (was: Ne

Re: [openstack-dev] [Nova] Ceilometer vs. Nova internal metrics collector for scheduling

2013-07-19 Thread Sandy Walsh
On 07/19/2013 12:30 PM, Sean Dague wrote: > On 07/19/2013 10:37 AM, Murray, Paul (HP Cloud Services) wrote: >> If we agree that "something like capabilities" should go through Nova, >> what do you suggest should be done with the change that sparked this >> debate: https://review.openstack.org/#/c

Re: [openstack-dev] A simple way to improve nova scheduler

2013-07-19 Thread Sandy Walsh
avlovic > > Mirantis Inc. > > > On Fri, Jul 19, 2013 at 11:47 PM, Sandy Walsh <mailto:sandy.wa...@rackspace.com>> wrote: > > > > On 07/19/2013 04:25 PM, Brian Schott wrote: > > I think Soren suggested this way back in Cactus to use MQ

Re: [openstack-dev] A simple way to improve nova scheduler

2013-07-19 Thread Sandy Walsh
On 07/19/2013 04:25 PM, Brian Schott wrote: > I think Soren suggested this way back in Cactus to use MQ for compute > node state rather than database and it was a good idea then. The problem with that approach was the number of queues went exponential as soon as you went beyond simple flavors.

Re: [openstack-dev] A simple way to improve nova scheduler

2013-07-19 Thread Sandy Walsh
not suitable and was eliminated a long time ago. > > > Best regards, > Boris Pavlovic > > Mirantis Inc. > > > > On Sat, Jul 20, 2013 at 12:23 AM, Sandy Walsh <mailto:sandy.wa...@rackspace.com>> wrote: > > > > On 07/19/2013 05:01 PM, Boris Pavl

Re: [openstack-dev] [Openstack] Ceilometer and notifications

2013-08-01 Thread Sandy Walsh
On 08/01/2013 07:02 AM, Mark McLoughlin wrote: > On Thu, 2013-08-01 at 10:36 +0200, Julien Danjou wrote: >> On Thu, Aug 01 2013, Sam Morrison wrote: >> >>> OK so is it that ceilometer just leaves the message on the queue or >>> only consumes certain messages? >> >> Ceilometer uses its own queue.

Re: [openstack-dev] Weight normalization in scheduler

2013-08-01 Thread Sandy Walsh
On 08/01/2013 04:24 AM, Álvaro López García wrote: > Hi all. > > TL;DR: I've created a blueprint [1] regarding weight normalization. > I would be very glad if somebody could examine and comment it. Something must have changed. It's been a while since I've done anything with the scheduler, but n

Re: [openstack-dev] [Openstack] Ceilometer and notifications

2013-08-01 Thread Sandy Walsh
On 08/01/2013 09:19 AM, Julien Danjou wrote: > On Thu, Aug 01 2013, Sandy Walsh wrote: > >> Hmm, if notifications are turned on, it should fill up. For billing >> purposes we don't want to lose events simply because there is no >> consumer. Operations would alert on

Re: [openstack-dev] Weight normalization in scheduler

2013-08-01 Thread Sandy Walsh
On 08/01/2013 09:51 AM, Álvaro López García wrote: > On Thu 01 Aug 2013 (09:07), Sandy Walsh wrote: >> >> >> On 08/01/2013 04:24 AM, Álvaro López García wrote: >>> Hi all. >>> >>> TL;DR: I've created a blueprint [1] regarding weight norm

Re: [openstack-dev] [Openstack] Ceilometer and notifications

2013-08-01 Thread Sandy Walsh
On 08/01/2013 10:25 AM, Julien Danjou wrote: > On Thu, Aug 01 2013, Sandy Walsh wrote: > >> Right, that is a concern. Within RAX we have two downstream services >> that consume notifications (StackTach and Yagi) and we've configured >> nova to write to two queue

[openstack-dev] [Ceilometer] Alarming should be outside of Ceilometer as a separate package.

2013-08-01 Thread Sandy Walsh
Hey y'all, I've had a little thorn in my claw on this topic for a while and thought I'd ask the larger group. I applaud the efforts of the people working on the alarming additions to Ceilometer, but I've got concerns that we're packaging things the wrong way. I fear we're making another Zenoss/N

Re: [openstack-dev] [Openstack] Ceilometer and notifications

2013-08-01 Thread Sandy Walsh
On 08/01/2013 07:25 PM, Doug Hellmann wrote: > > > > On Thu, Aug 1, 2013 at 9:25 AM, Julien Danjou <mailto:jul...@danjou.info>> wrote: > > On Thu, Aug 01 2013, Sandy Walsh wrote: > > > Right, that is a concern. Within RAX we have two down

Re: [openstack-dev] [Ceilometer] Alarming should be outside of Ceilometer as a separate package.

2013-08-01 Thread Sandy Walsh
On 08/01/2013 07:22 PM, Doug Hellmann wrote: > > > > On Thu, Aug 1, 2013 at 10:31 AM, Sandy Walsh <mailto:sandy.wa...@rackspace.com>> wrote: > > Hey y'all, > > I've had a little thorn in my claw on this topic for a while and thought

Re: [openstack-dev] [Ceilometer] Alarming should be outside of Ceilometer as a separate package.

2013-08-02 Thread Sandy Walsh
On 08/02/2013 08:38 AM, Doug Hellmann wrote: > > > > On Thu, Aug 1, 2013 at 8:52 PM, Sandy Walsh <mailto:sandy.wa...@rackspace.com>> wrote: > > > > On 08/01/2013 07:22 PM, Doug Hellmann wrote: > > > > > > &

Re: [openstack-dev] [Ceilometer] Alarming should be outside of Ceilometer as a separate package.

2013-08-02 Thread Sandy Walsh
On 08/02/2013 12:27 PM, Eoghan Glynn wrote: > >> On 08/01/2013 07:22 PM, Doug Hellmann wrote: >>> >>> >>> >>> On Thu, Aug 1, 2013 at 10:31 AM, Sandy Walsh >> <mailto:sandy.wa...@rackspace.com>> wrote: >>> >>>

Re: [openstack-dev] [Ceilometer] Alarming should be outside of Ceilometer as a separate package.

2013-08-05 Thread Sandy Walsh
On 08/04/2013 08:24 PM, Angus Salkeld wrote: > On 02/08/13 13:26 -0300, Sandy Walsh wrote: >> >> >> On 08/02/2013 12:27 PM, Eoghan Glynn wrote: >>> >>>> On 08/01/2013 07:22 PM, Doug Hellmann wrote: >>>>> >>>>> >>>

Re: [openstack-dev] [Ceilometer] Ceilometer and nova compute cells

2013-08-05 Thread Sandy Walsh
On 08/05/2013 04:49 AM, Julien Danjou wrote: > On Fri, Aug 02 2013, Doug Hellmann wrote: > >> On Fri, Aug 2, 2013 at 7:47 AM, Julien Danjou wrote: >>> That would need the RPC layer to connect to different rabbitmq server. >>> Not sure that's supported yet. >>> >> >> We'll have that problem in t

Re: [openstack-dev] [Ceilometer] Another introduction

2013-08-06 Thread Sandy Walsh
On 08/06/2013 08:31 AM, Nejc Saje wrote: > Hey everyone, > > I'm a developer at XLAB d.o.o. in Ljubljana. My colleagues are part of > the EU research project Contrail, dealing with cloud federation, and I > got hired to work on Contrail-Openstack integration. > > Firstly I'm trying to integrate

Re: [openstack-dev] [Heat] event table is a ticking time bomb

2013-08-09 Thread Sandy Walsh
On 08/08/2013 11:36 PM, Angus Salkeld wrote: > On 08/08/13 13:16 -0700, Clint Byrum wrote: >> Last night while reviewing a feature which would add more events to the >> event table, it dawned on me that the event table really must be removed. > > >> >> https://bugs.launchpad.net/heat/+bug/12094

Re: [openstack-dev] [glance] [ceilometer] Periodic Auditing In Glance

2013-08-13 Thread Sandy Walsh
On 08/13/2013 04:35 PM, Andrew Melton wrote: >> I'm just concerned with the type of notification you'd send. It has to >> be enough fine grained so we don't lose too much information. > > It's a tough situation, sending out an image.exists for each image with > the same payload as say image.uplo

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

2013-08-14 Thread Sandy Walsh
On 08/14/2013 06:17 AM, Julien Danjou wrote: > On Tue, Aug 13 2013, Thomas Maddox wrote: > > Hi Thomas, > >> * Driving get_resources() with the Meter table instead of the >> Resource table. This is mainly because of the additional filtering >> available in the Meter table, which allows

[openstack-dev] [oslo] ack(), reject() and requeue() support in rpc ...

2013-08-14 Thread Sandy Walsh
At Eric's request in https://review.openstack.org/#/c/41979/ I'm bringing this to the ML for feedback. Currently, oslo-common rpc behaviour is to always ack() a message no matter what. For billing purposes we can't afford to drop important notifications (like *.exists). We only want to ack() if n

[openstack-dev] [Ceilometer] Pipeline Retry Semantics ...

2013-08-15 Thread Sandy Walsh
Recently I've been focused on ensuring we don't drop notifications in CM. But problems still exist downstream, after we've captured the raw event. >From the efforts going on with the Ceilometer sample pipeline, the new dispatcher model and the upcoming trigger pipeline, the discussion around retry

Re: [openstack-dev] [oslo] ack(), reject() and requeue() support in rpc ...

2013-08-15 Thread Sandy Walsh
On 08/15/2013 02:00 PM, Eric Windisch wrote: > On Wed, Aug 14, 2013 at 4:08 PM, Sandy Walsh > wrote: >> At Eric's request in https://review.openstack.org/#/c/41979/ I'm >> bringing this to the ML for feedback. > > Thank you Sandy. > >> Currently,

Re: [openstack-dev] [oslo] ack(), reject() and requeue() support in rpc ...

2013-08-16 Thread Sandy Walsh
On 08/16/2013 09:47 AM, Flavio Percoco wrote: > On 14/08/13 17:08 -0300, Sandy Walsh wrote: >> At Eric's request in https://review.openstack.org/#/c/41979/ I'm >> bringing this to the ML for feedback. >> >> Currently, oslo-common rpc behaviour is to alw

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

2013-08-19 Thread Sandy Walsh
On 08/19/2013 05:08 AM, Julien Danjou wrote: > On Sun, Aug 18 2013, Jay Pipes wrote: > >> I'm proposing that in these cases, a *new* resource would be added to the >> resource table (and its ID inserted in meter) table with the new >> flavor/instance's metadata. > > Ah I see. Considering we're

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

2013-08-19 Thread Sandy Walsh
On 08/18/2013 04:04 PM, Jay Pipes wrote: > On 08/17/2013 03:10 AM, Julien Danjou wrote: >> On Fri, Aug 16 2013, Jay Pipes wrote: >> >>> Actually, that's the opposite of what I'm suggesting :) I'm suggesting >>> getting rid of the resource_metadata column in the meter table and >>> using the >>> r

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

2013-08-19 Thread Sandy Walsh
On 08/19/2013 09:40 AM, Julien Danjou wrote: > On Mon, Aug 19 2013, Sandy Walsh wrote: > >> On 08/19/2013 05:08 AM, Julien Danjou wrote: >>> On Sun, Aug 18 2013, Jay Pipes wrote: >>> >>>> I'm proposing that in these cases, a *new* resource would

Re: [openstack-dev] [glance] [ceilometer] Periodic Auditing In Glance

2013-08-19 Thread Sandy Walsh
On 08/16/2013 04:58 PM, Doug Hellmann wrote: > The notification messages don't translate 1:1 to database records. Even > if the notification payload includes multiple resources, we will store > those as multiple individual records so we can query against them. So it > seems like sending individua

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

2013-08-20 Thread Sandy Walsh
On 08/20/2013 10:42 AM, Thomas Maddox wrote: > On 8/19/13 8:21 AM, "Sandy Walsh" wrote: > >> >> >> On 08/18/2013 04:04 PM, Jay Pipes wrote: >>> On 08/17/2013 03:10 AM, Julien Danjou wrote: >>>> On Fri, Aug 16 2013, Jay Pipes wrot

Re: [openstack-dev] [Ceilometer] Need help with Alembic...

2013-08-26 Thread Sandy Walsh
I'm getting the same problem with a different migration (mine is complaining that a column already exists) http://paste.openstack.org/show/44512/ I've compared it to the other migrations and it seems fine. -S On 08/26/2013 02:34 PM, Jay Pipes wrote: > Hey all, > > I'm trying to figure out what

Re: [openstack-dev] [Ceilometer] Need help with Alembic...

2013-08-27 Thread Sandy Walsh
s in the > current source code base using a call to > sqlalchemy.MetaData.create___all(). > > This results in re-running migrations over and over again, > instead of > having dedicated migration test

Re: [openstack-dev] [Nova] Frustrations with review wait times

2013-08-28 Thread Sandy Walsh
On 08/28/2013 10:58 AM, Thierry Carrez wrote: > Robert Collins wrote: >> So I'd like to throw two ideas into the mix. >> >> Firstly, consider having a rota - ideally 24x5 but that will need some >> more geographical coverage I suspect for many projects - of folk who >> spend a dedicated time peri

Re: [openstack-dev] [Ceilometer] what's in scope of Ceilometer

2013-08-29 Thread Sandy Walsh
On 08/29/2013 05:20 AM, Julien Danjou wrote: > On Thu, Aug 29 2013, Gordon Chung wrote: > >> the first question is, Ceilometer currently does metering/alarming/maybe a >> few other things... will it go beyond that? specifically: capacity >> planning, optimization, dashboard(i assume this falls

Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-30 Thread Sandy Walsh
> >From: Johannes Erdfelt [johan...@erdfelt.com] >Sent: Thursday, January 29, 2015 9:18 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues > >On Thu, Jan 29, 2

Re: [openstack-dev] [oslo.messaging] extending notification MessagingDriver

2015-02-26 Thread Sandy Walsh
Cool, I'm interested in creating some notification drivers outside of olso-messaging as well (for Kafka support and schema-based notifications). Do you have a repo started on this? I'd be keen to have a look. -S From: Boden Russell Sent: Thursday, Febr

[openstack-dev] StackTach.v3 docs ...

2015-02-26 Thread Sandy Walsh
First pass at new docs are available here http://www.stacktach.com/ (API and glossary to follow) Feedback and patches welcome! -Sandy __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-de

Re: [openstack-dev] [oslo_messaging]Notification listener for nova

2015-03-04 Thread Sandy Walsh
I assume you have notifications enabled in nova.conf? See the "Enabling notifications in OpenStack" section http://www.stacktach.com/about.html Oh, there are couple more flags that could be useful too: notify_on_state_change=vm_and_task_state notify_on_any_change=True ? __

Re: [openstack-dev] [stacktach] [oslo] stachtach -> kombu -> pika ??

2015-03-10 Thread Sandy Walsh
Hey J, Our (old) notification consumer was using carrot, which is dead but worked. Lately though there have been conflicts with carrot and msgpack, so we had to change. Around the same time, we ran into a bug where we were writing to an unnamed exchange (completely valid, but too easy to do und

Re: [openstack-dev] [stacktach] [oslo] stachtach -> kombu -> pika ??

2015-03-10 Thread Sandy Walsh
No, we're adding this to Yagi first and perhaps Notabene later. We don't need rpc support, so it's too big a change for us to take on. From: gordon chung Sent: Tuesday, March 10, 2015 3:58 PM To: OpenStack Development Mailing List not for usage questions Subjec

Re: [openstack-dev] [stacktach] [oslo] stachtach -> kombu -> pika ??

2015-03-16 Thread Sandy Walsh
> >From: Sean Dague >Sent: Thursday, March 12, 2015 9:09 AM >To: openstack-dev@lists.openstack.org >Subject: Re: [openstack-dev] [stacktach] [oslo] stachtach -> kombu -> pika ?? > >On 03/10/2015 12:56 PM, Joshua Harlow wrote: >> I saw the following on >> htt

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-07 Thread Sandy Walsh
Notifications were added for this very purpose. At Rax, we have a downstream consumer (yagi) that routes notifications to an appropriate ATOM/pubsub feed (based on tenant-id). Notifications are only as heavy as you want to make them. The only required fields are event_type and timestamp. ?

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-07 Thread Sandy Walsh
>Tooling in general seems to be moving towards richer event data as well. >The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are >intended to take your unstructured logs and turn them into events, so >why not have Heat output structured events that we can present to the >user with

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-08 Thread Sandy Walsh
>From: Ryan Brown >Sent: Wednesday, April 8, 2015 9:42 AM > >> The trend in the monitoring space seems to be: >> >> 1. Alarms are issued from Metrics as Events. >> (events can issue alarms too, but conventional alarming is metric based) >> 2. Multiple events are analyzed to produce Metrics (st

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-08 Thread Sandy Walsh
>From: Clint Byrum >Sent: Wednesday, April 8, 2015 1:15 PM > >There's this: > >https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. >I also want to point out that what I'd actually rather see is that all >of the services provide fun

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-08 Thread Sandy Walsh
Yeah, I don't think anyone would give access to the production rabbitmq directly. We use Yagi [1] to pipe it to AtomHopper [2] for downstream consumption/sanitizing. -S [1] https://github.com/rackerlabs/yagi [2] http://atomhopper.org/ From: Duncan Thomas

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-09 Thread Sandy Walsh
>From: Angus Salkeld >Sent: Wednesday, April 8, 2015 8:24 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [all] how to send messages (and events) to our >users > > >I also want to point out that what I'd actually rather see is that all >of the se

Re: [openstack-dev] [nova][oslo_messaging]Enabling notifications from nova

2015-04-16 Thread Sandy Walsh
http://www.stacktach.com/about.html#enabling? From: Eduard Matei Sent: Thursday, April 16, 2015 6:04 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova][oslo_messaging]Enabling notifications from nova Hi, I've

[openstack-dev] StackTach.v3 now in production ...

2015-04-22 Thread Sandy Walsh
(sorry for cross-post, but this is appropriate to both audiences) Hey y'all! For those of you that don't know, StackTach is a notification-based debugging, monitoring and usage tool for OpenStack. We're happy to announce that we've recently rolled StackTach.v3 into production at one of the Rax

<    1   2