Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Sandy Walsh]

On 08/01/2013 07:25 PM, Doug Hellmann wrote:
> 
> 
> 
> On Thu, Aug 1, 2013 at 9:25 AM, Julien Danjou  > wrote:
> 
> On Thu, Aug 01 2013, Sandy Walsh wrote:
> 
> > Right, that is a concern. Within RAX we have two downstream services
> > that consume notifications (StackTach and Yagi) and we've configured
> > nova to write to two queues. --notifications_topics can take a list.
> 
> Fair enough. So IIUC that means we should change Ceilometer to consume
> CONF.notification_topics using the default queue associated with this
> topic, and let the deployer add more notification_topics in nova if they
> have several consumer (that should have their own
> CONF.notification_topics).
> 
> This would be the correct way, even if not really deployer-friendly.
> 
> 
> The other solution would be to have the sender of the message NOT
> declare a queue at all. The queue is a client-side feature, and the
> sender shouldn't care about the queue.

But that's the problem. Once openstack is running (and making money) we
don't want to lose any events. So we have to create the queue on the
producer side. Normally, yes, the consumer should create the queue.

> 
> Doug
>  
> 
> 
> > I don't think so, but if that was possible it would solve our problem.
> >
> > AFAIK, amqp only uses the exchange as a dispatcher and all storage is
> > done in the queue ... but I could be wrong. I vaguely recall there
> being
> > a durable exchange setting as well as durable queue.
> >
> > I'll do some investigating.
> 
> Yeah I've little hope too, but might worth a look. Thanks Sandy!
> 
> --
> Julien Danjou
> # Free Software hacker # freelance consultant
> # http://julien.danjou.info
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Doug Hellmann]

On Thu, Aug 1, 2013 at 9:25 AM, Julien Danjou  wrote:

> On Thu, Aug 01 2013, Sandy Walsh wrote:
>
> > Right, that is a concern. Within RAX we have two downstream services
> > that consume notifications (StackTach and Yagi) and we've configured
> > nova to write to two queues. --notifications_topics can take a list.
>
> Fair enough. So IIUC that means we should change Ceilometer to consume
> CONF.notification_topics using the default queue associated with this
> topic, and let the deployer add more notification_topics in nova if they
> have several consumer (that should have their own
> CONF.notification_topics).
>
> This would be the correct way, even if not really deployer-friendly.
>

The other solution would be to have the sender of the message NOT declare a
queue at all. The queue is a client-side feature, and the sender shouldn't
care about the queue.

Doug


>
> > I don't think so, but if that was possible it would solve our problem.
> >
> > AFAIK, amqp only uses the exchange as a dispatcher and all storage is
> > done in the queue ... but I could be wrong. I vaguely recall there being
> > a durable exchange setting as well as durable queue.
> >
> > I'll do some investigating.
>
> Yeah I've little hope too, but might worth a look. Thanks Sandy!
>
> --
> Julien Danjou
> # Free Software hacker # freelance consultant
> # http://julien.danjou.info
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Rosa, Andrea (HP Cloud Services)]

HI 

>AFAIK, amqp only uses the exchange as a dispatcher and all storage is done in
>the queue ... but I could be wrong. I vaguely recall there being a durable
>exchange setting as well as durable queue.

AFAIK if a message can't be routed there are three options:
 - the message is discarded (I think this is the default option)
-  the message is sent back to the publisher (some params need to be specified 
when a message is published)
- some broker can have specific queues for those messages, for example Rabbitmq 
has a "dead letter queue" where not routable messages are stored.

Durable exchanges are exchanges that survive to a server restart, they don't 
need to be re-declared.
HTH
--
Andrea Rosa

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Sandy Walsh]

On 08/01/2013 10:25 AM, Julien Danjou wrote:
> On Thu, Aug 01 2013, Sandy Walsh wrote:
> 
>> Right, that is a concern. Within RAX we have two downstream services
>> that consume notifications (StackTach and Yagi) and we've configured
>> nova to write to two queues. --notifications_topics can take a list.
> 
> Fair enough. So IIUC that means we should change Ceilometer to consume
> CONF.notification_topics using the default queue associated with this
> topic, and let the deployer add more notification_topics in nova if they
> have several consumer (that should have their own
> CONF.notification_topics).
> 
> This would be the correct way, even if not really deployer-friendly.

There are risks, but I think it'll offer a better "out-of-the-box"
experience.

> 
>> I don't think so, but if that was possible it would solve our problem.
>>
>> AFAIK, amqp only uses the exchange as a dispatcher and all storage is
>> done in the queue ... but I could be wrong. I vaguely recall there being
>> a durable exchange setting as well as durable queue.
>>
>> I'll do some investigating.
> 
> Yeah I've little hope too, but might worth a look. Thanks Sandy!
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Julien Danjou]

On Thu, Aug 01 2013, Sandy Walsh wrote:

> Right, that is a concern. Within RAX we have two downstream services
> that consume notifications (StackTach and Yagi) and we've configured
> nova to write to two queues. --notifications_topics can take a list.

Fair enough. So IIUC that means we should change Ceilometer to consume
CONF.notification_topics using the default queue associated with this
topic, and let the deployer add more notification_topics in nova if they
have several consumer (that should have their own
CONF.notification_topics).

This would be the correct way, even if not really deployer-friendly.

> I don't think so, but if that was possible it would solve our problem.
>
> AFAIK, amqp only uses the exchange as a dispatcher and all storage is
> done in the queue ... but I could be wrong. I vaguely recall there being
> a durable exchange setting as well as durable queue.
>
> I'll do some investigating.

Yeah I've little hope too, but might worth a look. Thanks Sandy!

-- 
Julien Danjou
# Free Software hacker # freelance consultant
# http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Sandy Walsh]

On 08/01/2013 09:19 AM, Julien Danjou wrote:
> On Thu, Aug 01 2013, Sandy Walsh wrote:
> 
>> Hmm, if notifications are turned on, it should fill up. For billing
>> purposes we don't want to lose events simply because there is no
>> consumer. Operations would alert on it and someone would need to put out
>> the fire.
> 
> So currently, are we're possibly losing events because we don't use the
> standard queue but one defined by Ceilometer upon connection?
> We can't consume events from the default notifications queue or we would
> break any tool possibly using it. Each tool consuming needs a copy of
> them.

Right, that is a concern. Within RAX we have two downstream services
that consume notifications (StackTach and Yagi) and we've configured
nova to write to two queues. --notifications_topics can take a list.

> Isn't there a way to queue the message in exchanges if there's no queue
> at all?

I don't think so, but if that was possible it would solve our problem.

AFAIK, amqp only uses the exchange as a dispatcher and all storage is
done in the queue ... but I could be wrong. I vaguely recall there being
a durable exchange setting as well as durable queue.

I'll do some investigating.

> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Julien Danjou]

On Thu, Aug 01 2013, Sandy Walsh wrote:

> Hmm, if notifications are turned on, it should fill up. For billing
> purposes we don't want to lose events simply because there is no
> consumer. Operations would alert on it and someone would need to put out
> the fire.

So currently, are we're possibly losing events because we don't use the
standard queue but one defined by Ceilometer upon connection?
We can't consume events from the default notifications queue or we would
break any tool possibly using it. Each tool consuming needs a copy of
them.

Isn't there a way to queue the message in exchanges if there's no queue
at all?

-- 
Julien Danjou
-- Free Software hacker - freelance consultant
-- http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Sandy Walsh]

On 08/01/2013 07:02 AM, Mark McLoughlin wrote:
> On Thu, 2013-08-01 at 10:36 +0200, Julien Danjou wrote:
>> On Thu, Aug 01 2013, Sam Morrison wrote:
>>
>>> OK so is it that ceilometer just leaves the message on the queue or
>>> only consumes certain messages?
>>
>> Ceilometer uses its own queue. There might be other processes consuming
>> this notifications, so removing them may be not a good idea.
>>
>> The problem may be that the notification sender create a queue by
>> default even if there's no consumer on that. Maybe that's something we
>> should avoid doing in Oslo (Cc'ing -dev to get advice on that).
> 
> I'm missing the context here, but it sounds like the default
> notifications queue created isn't the one consumed by ceilometer so it
> fills up and we just shouldn't be creating that queue.
> 
> Sounds reasonable to me. Definitely file a bug for it.

Hmm, if notifications are turned on, it should fill up. For billing
purposes we don't want to lose events simply because there is no
consumer. Operations would alert on it and someone would need to put out
the fire.

That's the reason we create the queue up front in the first place.
Ideally, we could only write to the exchange, but we need the queue to
ensure we don't lose any events.

The CM Collector consumes from two queues: it's internal queue and the
Nova queue (if configured). If CM is looking at the wrong nova queue by
default, the bug would be over there.


> 
> Cheers,
> Mark.
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Mark McLoughlin]

On Thu, 2013-08-01 at 10:36 +0200, Julien Danjou wrote:
> On Thu, Aug 01 2013, Sam Morrison wrote:
> 
> > OK so is it that ceilometer just leaves the message on the queue or
> > only consumes certain messages?
> 
> Ceilometer uses its own queue. There might be other processes consuming
> this notifications, so removing them may be not a good idea.
> 
> The problem may be that the notification sender create a queue by
> default even if there's no consumer on that. Maybe that's something we
> should avoid doing in Oslo (Cc'ing -dev to get advice on that).

I'm missing the context here, but it sounds like the default
notifications queue created isn't the one consumed by ceilometer so it
fills up and we just shouldn't be creating that queue.

Sounds reasonable to me. Definitely file a bug for it.

Cheers,
Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] Ceilometer and notifications

[Julien Danjou]

On Thu, Aug 01 2013, Sam Morrison wrote:

> OK so is it that ceilometer just leaves the message on the queue or
> only consumes certain messages?

Ceilometer uses its own queue. There might be other processes consuming
this notifications, so removing them may be not a good idea.

The problem may be that the notification sender create a queue by
default even if there's no consumer on that. Maybe that's something we
should avoid doing in Oslo (Cc'ing -dev to get advice on that).

-- 
Julien Danjou
-- Free Software hacker - freelance consultant
-- http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev