Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2014-01-03 Thread Gordon Sim
On 01/02/2014 10:46 PM, Herndon, John Luke wrote: On 1/2/14, 11:36 AM, Gordon Sim g...@redhat.com wrote: On 12/20/2013 09:26 PM, Herndon, John Luke wrote: On Dec 20, 2013, at 12:13 PM, Gordon Sim g...@redhat.com wrote: On 12/20/2013 05:27 PM, Herndon, John Luke wrote: Other protocols

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2014-01-02 Thread Herndon, John Luke
On 12/20/13, 11:57 PM, Jay Pipes jaypi...@gmail.com wrote: On 12/20/2013 04:43 PM, Julien Danjou wrote: On Fri, Dec 20 2013, Herndon, John Luke wrote: I think there is probably a tolerance for duplicates but you¹re right, missing a notification is unacceptable. Can anyone weigh in on how

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2014-01-02 Thread Gordon Sim
On 12/20/2013 09:26 PM, Herndon, John Luke wrote: On Dec 20, 2013, at 12:13 PM, Gordon Sim g...@redhat.com wrote: On 12/20/2013 05:27 PM, Herndon, John Luke wrote: Other protocols may support bulk consumption. My one concern with this approach is error handling. Currently the executors

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2014-01-02 Thread Herndon, John Luke
On 1/2/14, 11:36 AM, Gordon Sim g...@redhat.com wrote: On 12/20/2013 09:26 PM, Herndon, John Luke wrote: On Dec 20, 2013, at 12:13 PM, Gordon Sim g...@redhat.com wrote: On 12/20/2013 05:27 PM, Herndon, John Luke wrote: Other protocols may support bulk consumption. My one concern with

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-21 Thread Jay Pipes
On 12/21/2013 04:51 AM, Boris Pavlovic wrote: Jay, The session object will be commit()'d after the session.begin() context manager exits, which will cause the aforementioned BEGIN; INSERT; COMMIT; transaction to be executed against the server for each event record. It is a just half of

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Nadya Privalova
Hi John, As for me your ideas look very interesting. As I understood notification messages will be kept in MQ for some time (during batch-basket is being filled), right? I'm concerned about the additional load that will be on MQ (Rabbit). Thanks, Nadya On Fri, Dec 20, 2013 at 3:31 AM, Herndon,

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Herndon, John Luke
@lists.openstack.org Date: Fri, 20 Dec 2013 15:25:55 +0400 To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches Hi John, As for me your ideas look very interesting. As I

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Doug Hellmann
On Thu, Dec 19, 2013 at 6:31 PM, Herndon, John Luke john.hern...@hp.comwrote: Hi Folks, The Rackspace-HP team has been putting a lot of effort into performance testing event collection in the ceilometer storage drivers[0]. Based on some results of this testing, we would like to support batch

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Julien Danjou
On Thu, Dec 19 2013, Herndon, John Luke wrote: Hi John, The Rackspace-HP team has been putting a lot of effort into performance testing event collection in the ceilometer storage drivers[0]. Based on some results of this testing, we would like to support batch consumption of notifications,

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Herndon, John Luke
On Dec 20, 2013, at 8:10 AM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Thu, Dec 19, 2013 at 6:31 PM, Herndon, John Luke john.hern...@hp.com wrote: Hi Folks, The Rackspace-HP team has been putting a lot of effort into performance testing event collection in the ceilometer

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Herndon, John Luke
On Dec 20, 2013, at 8:48 AM, Julien Danjou jul...@danjou.info wrote: On Thu, Dec 19 2013, Herndon, John Luke wrote: Hi John, The Rackspace-HP team has been putting a lot of effort into performance testing event collection in the ceilometer storage drivers[0]. Based on some results of

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Julien Danjou
On Fri, Dec 20 2013, Herndon, John Luke wrote: Yeah, I like this idea. As far as I can tell, AMQP doesn’t support grabbing more than a single message at a time, but we could definitely have the broker store up the batch before sending it along. Other protocols may support bulk consumption. My

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Herndon, John Luke
On Dec 20, 2013, at 10:47 AM, Julien Danjou jul...@danjou.info wrote: On Fri, Dec 20 2013, Herndon, John Luke wrote: Yeah, I like this idea. As far as I can tell, AMQP doesn’t support grabbing more than a single message at a time, but we could definitely have the broker store up the batch

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Gordon Sim
On 12/20/2013 05:27 PM, Herndon, John Luke wrote: On Dec 20, 2013, at 8:48 AM, Julien Danjou jul...@danjou.info wrote: Anyway, my main concern here is that I am not very enthusiast about using the executor to do that. I wonder if there is not a way to ask the broker to get as many as message

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Gordon Sim
On 12/20/2013 07:13 PM, Gordon Sim wrote: AMQP (in all it's versions) allows for a subscription with a configurable amount of 'prefetch', which means the broker can send lots of messages without waiting for the client to request them one at a time. Just as an aside, the impl_qpid.py driver

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Dan Dyer
On 12/20/2013 11:18 AM, Herndon, John Luke wrote: On Dec 20, 2013, at 10:47 AM, Julien Danjou jul...@danjou.info wrote: On Fri, Dec 20 2013, Herndon, John Luke wrote: Yeah, I like this idea. As far as I can tell, AMQP doesn't support grabbing more than a single message at a time, but we

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Doug Hellmann
On Fri, Dec 20, 2013 at 12:15 PM, Herndon, John Luke john.hern...@hp.comwrote: On Dec 20, 2013, at 8:10 AM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Thu, Dec 19, 2013 at 6:31 PM, Herndon, John Luke john.hern...@hp.comwrote: Hi Folks, The Rackspace-HP team has been putting

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Herndon, John Luke
On Dec 20, 2013, at 12:13 PM, Gordon Sim g...@redhat.com wrote: On 12/20/2013 05:27 PM, Herndon, John Luke wrote: On Dec 20, 2013, at 8:48 AM, Julien Danjou jul...@danjou.info wrote: Anyway, my main concern here is that I am not very enthusiast about using the executor to do that. I

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Julien Danjou
On Fri, Dec 20 2013, Herndon, John Luke wrote: I think there is probably a tolerance for duplicates but you’re right, missing a notification is unacceptable. Can anyone weigh in on how big of a deal duplicates are for meters? Duplicates aren’t really unique to the batching approach, though.

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Herndon, John Luke
On Dec 20, 2013, at 1:12 PM, Dan Dyer dan.dye...@gmail.com wrote: On 12/20/2013 11:18 AM, Herndon, John Luke wrote: On Dec 20, 2013, at 10:47 AM, Julien Danjou jul...@danjou.info wrote: On Fri, Dec 20 2013, Herndon, John Luke wrote: Yeah, I like this idea. As far as I can tell, AMQP

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Julien Danjou
On Fri, Dec 20 2013, Herndon, John Luke wrote: I think this will work. I was considering putting in a timeout so the broker would not send off all of the messages immediately, and implement using blocking calls. If the consumer consumes faster than the publishers are publishing, this just

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-20 Thread Jay Pipes
On 12/20/2013 04:43 PM, Julien Danjou wrote: On Fri, Dec 20 2013, Herndon, John Luke wrote: I think there is probably a tolerance for duplicates but you’re right, missing a notification is unacceptable. Can anyone weigh in on how big of a deal duplicates are for meters? Duplicates aren’t

[openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

2013-12-19 Thread Herndon, John Luke
Hi Folks, The Rackspace-HP team has been putting a lot of effort into performance testing event collection in the ceilometer storage drivers[0]. Based on some results of this testing, we would like to support batch consumption of notifications, as it will greatly improve insertion performance.