ceilometer is going to need to subscribe several worker processes to the notifications.info topic for the other services like nova, glance, and quantum. The pool of workers needs to be assured of receiving all messages, without interference from other clients listening for notifications (such as metering, audit logging, monitoring, etc.).
The drivers implement create_consumer() for topic consumers by always making the topic and queue name the same. That supports the consumer patterns we have encountered so far, but does not allow for load balancing between workers as we want for the ceilometer collector. The fanout argument is used by the existing drivers to control the breadth of distribution. With fanout=True, every consumer receives a copy of the event because every consumer has its own queue. This gives us no load balancing benefit for having multiple consumers. With fanout=False only one consumer *at all* will get a copy of the event, since they all listen on the same queue. This means if metering, audit logging, and monitoring are all listening for notifications they will each see only some of the events. The way to achieve load balancing is to have the ceilometer consumers connect to the same exchange and topic using a shared, well-known, queue name that is different from the name used by non-ceilometer consumers. Unfortunately, the only parameter that cannot be controlled by the caller of create_consumer() is the queue name. I have a patch ready for review [1] to add a new method, create_worker() to the RPC Connection class, to allow a group of consumer to share a queue to manage load. The worker pool allows multiple consumers to receive a given message (by subscribing to separate queues), but it also allows several consumers to declare that they are collaborating so that only one of the subset receives a copy (by subscribing to the same queue). That means that multiple "types" of consumers can be listening to notifications and each type of consumer can have a load balanced pool of workers so that messages are only processed once for that type (once for metering and once for logging, for example). Two separate implementations were discussed: Adding a "queue_name" argument to create_consumer() and creating a new method "create_worker()". After considering both options, I chose to add a new method because it clarified the right way to combine the inputs to set up workers (fanout must always be true and the queue name must always be provided). The code is up for review, so please have a look and let me know what you think. Doug [1] https://review.openstack.org/#/c/7590/
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
