On Fri, Dec 20, 2013 at 12:15 PM, Herndon, John Luke <john.hern...@hp.com>wrote:
> > On Dec 20, 2013, at 8:10 AM, Doug Hellmann <doug.hellm...@dreamhost.com> > wrote: > > > > > On Thu, Dec 19, 2013 at 6:31 PM, Herndon, John Luke > <john.hern...@hp.com>wrote: > >> Hi Folks, >> >> The Rackspace-HP team has been putting a lot of effort into performance >> testing event collection in the ceilometer storage drivers[0]. Based on >> some results of this testing, we would like to support batch consumption >> of notifications, as it will greatly improve insertion performance. Batch >> consumption in this case means waiting for a certain number of >> notifications to arrive before sending to the storage >> driver. >> >> I¹d like to get feedback from the community about this feature, and how we >> are planning to implement it. Here is what I’m currently thinking: >> >> 1) This seems to fit well into oslo.messaging - batching may be a feature >> that other projects will find useful. After reviewing the changes that >> sileht has been working on in oslo.messaging, I think the right way to >> start off is to create a new executor that builds up a batch of >> notifications, and sends the batch to the dispatcher. We’d also add a >> timeout, so if a certain amount of time passes and the batch isn’t filled >> up, the notifications will be dispatched anyway. I’ve started a >> blueprint for this change and am filling in the details as I go along [1]. >> > > IIRC, the executor is meant to differentiate between threading, eventlet, > other async implementations, or other methods for dealing with the I/O. It > might be better to implement the batching at the dispatcher level instead. > That way no matter what I/O processing is in place, the batching will occur. > > > I thought about doing it in the dispatcher. One problem I see is handling > message acks. It looks like the current executors are built around single > messages andre-queueing single messages if problems occur. If we build up a > batch in the dispatcher, either the executor has to wait for the whole > batch to be committed (which wouldn’t work in the case of the blocking > executor, or would leave a lot of green threads hanging around in the case > of the eventlet executor), or the executor has to be modified to allow > acking to be handled out of band. So, I was thinking it would be cleaner to > write a new executor that is responsible for acking/requeueing the entire > batch. Maybe I’m missing something? > No, you're right. Were you going to use eventlet again for the new executor? > > >> 2) In ceilometer, initialize the notification listener with the batch >> executor instead of the eventlet executor (this should probably be >> configurable)[2]. We can then send the entire batch of notifications to >> the storage driver to be processed as events, while maintaining the >> current method for converting notifications into samples. >> >> 3) Error handling becomes more difficult. The executor needs to know if >> any of the notifications should be requeued. I think the right way to >> solve this is to return a list of notifications to requeue from the >> handler. Any better ideas? >> > > Which "handler" do you mean? > > > Ah, sorry - handler is whichever method is registered to receive the batch > from the dispatcher. In ceilometer’s case, this would be > process_notifications I think. > > > Doug > > > >> >> Is this the right approach to take? I¹m not an oslo.messaging expert, so >> if there is a proper way to implement this change, I¹m all ears! >> >> Thanks, happy holidays! >> -john >> >> 0: https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing >> 1: >> >> https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages >> 2: >> https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ----------------- > John Herndon > HP Cloud > john.hern...@hp.com > > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev