On Dec 20, 2013, at 8:10 AM, Doug Hellmann <[email protected]> wrote:
> > > > On Thu, Dec 19, 2013 at 6:31 PM, Herndon, John Luke <[email protected]> > wrote: > Hi Folks, > > The Rackspace-HP team has been putting a lot of effort into performance > testing event collection in the ceilometer storage drivers[0]. Based on > some results of this testing, we would like to support batch consumption > of notifications, as it will greatly improve insertion performance. Batch > consumption in this case means waiting for a certain number of > notifications to arrive before sending to the storage > driver. > > I¹d like to get feedback from the community about this feature, and how we > are planning to implement it. Here is what I’m currently thinking: > > 1) This seems to fit well into oslo.messaging - batching may be a feature > that other projects will find useful. After reviewing the changes that > sileht has been working on in oslo.messaging, I think the right way to > start off is to create a new executor that builds up a batch of > notifications, and sends the batch to the dispatcher. We’d also add a > timeout, so if a certain amount of time passes and the batch isn’t filled > up, the notifications will be dispatched anyway. I’ve started a > blueprint for this change and am filling in the details as I go along [1]. > > IIRC, the executor is meant to differentiate between threading, eventlet, > other async implementations, or other methods for dealing with the I/O. It > might be better to implement the batching at the dispatcher level instead. > That way no matter what I/O processing is in place, the batching will occur. > I thought about doing it in the dispatcher. One problem I see is handling message acks. It looks like the current executors are built around single messages andre-queueing single messages if problems occur. If we build up a batch in the dispatcher, either the executor has to wait for the whole batch to be committed (which wouldn’t work in the case of the blocking executor, or would leave a lot of green threads hanging around in the case of the eventlet executor), or the executor has to be modified to allow acking to be handled out of band. So, I was thinking it would be cleaner to write a new executor that is responsible for acking/requeueing the entire batch. Maybe I’m missing something? > > 2) In ceilometer, initialize the notification listener with the batch > executor instead of the eventlet executor (this should probably be > configurable)[2]. We can then send the entire batch of notifications to > the storage driver to be processed as events, while maintaining the > current method for converting notifications into samples. > > 3) Error handling becomes more difficult. The executor needs to know if > any of the notifications should be requeued. I think the right way to > solve this is to return a list of notifications to requeue from the > handler. Any better ideas? > > Which "handler" do you mean? Ah, sorry - handler is whichever method is registered to receive the batch from the dispatcher. In ceilometer’s case, this would be process_notifications I think. > Doug > > > > Is this the right approach to take? I¹m not an oslo.messaging expert, so > if there is a proper way to implement this change, I¹m all ears! > > Thanks, happy holidays! > -john > > 0: https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing > 1: > https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages > 2: https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ----------------- John Herndon HP Cloud [email protected]
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
