On Mon, Aug 19, 2013 at 3:06 PM, Sandy Walsh <[email protected]>wrote:
> > > On 08/16/2013 04:58 PM, Doug Hellmann wrote: > > The notification messages don't translate 1:1 to database records. Even > > if the notification payload includes multiple resources, we will store > > those as multiple individual records so we can query against them. So it > > seems like sending individual notifications would let us distribute the > > load of processing the notifications across several collector instances, > > and won't have any effect on the data storage requirements. > > > Well, they would. Each .exists would result in a Event record being > stored with the underlying raw json (pretty big) at the very least. > Ah, you're right, I forgot about the new event table. I was just thinking of the samples. Doug > > Like Alex said, if each customer has a daily week-long rolling backup > over 100k instances that's 700k .exists records. > > We have some ideas we're kicking around internally about alternative > approaches, but right now I think we have to design for 1 glance .exists > per image (worst case) or 1 glance event per tenant (better case) and > hope that deploying per-cell will help spread the load ... but it'll > suck for making aggregated reports per region. > > Phil, like Doug said, I don't think switching from per-instance to > per-tenant or anything else will really affect the end result. The > event->meter mapping will have to break it down anyway. > > > -S > > > > > Doug > > > > > > On Thu, Aug 15, 2013 at 11:58 AM, Alex Meade <[email protected] > > <mailto:[email protected]>> wrote: > > > > I don't know any actual numbers but I would have the concern that > > images tend to stick around longer than instances. For example, if > > someone takes daily snapshots of their server and keeps them around > > for a long time, the number of exists events would go up and up. > > > > Just a thought, could be a valid avenue of concern. > > > > -Alex > > > > -----Original Message----- > > From: "Doug Hellmann" <[email protected] > > <mailto:[email protected]>> > > Sent: Thursday, August 15, 2013 11:17am > > To: "OpenStack Development Mailing List" > > <[email protected] > > <mailto:[email protected]>> > > Subject: Re: [openstack-dev] [glance] [ceilometer] Periodic Auditing > > In Glance > > > > _______________________________________________ > > OpenStack-dev mailing list > > [email protected] > > <mailto:[email protected]> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > Nova generates a single exists event for each instance, and that > doesn't > > cause a lot of trouble as far as I've been able to see. > > > > What is the relative number of images compared to instances in a > > "typical" > > cloud? > > > > Doug > > > > > > On Tue, Aug 13, 2013 at 7:20 PM, Neal, Phil <[email protected] > > <mailto:[email protected]>> wrote: > > > > > I'm a little concerned that a batch payload won't align with > "exists" > > > events generated from other services. To my recollection, Cinder, > > Trove and > > > Neutron all emit exists events on a per-instance basis....a > > consumer would > > > have to figure out a way to handle/unpack these separately if they > > needed a > > > granular feed. Not the end of the world, I suppose, but a bit > > inconsistent. > > > > > > And a minor quibble: batching would also make it a much bigger > > issue if a > > > consumer missed a notification....though I guess you could > > counteract that > > > by increasing the frequency (but wouldn't that defeat the purpose?) > > > > > > > > > > > > > > > > > > > On 08/13/2013 04:35 PM, Andrew Melton wrote: > > > > >> I'm just concerned with the type of notification you'd send. > > It has to > > > > >> be enough fine grained so we don't lose too much information. > > > > > > > > > > It's a tough situation, sending out an image.exists for each > > image with > > > > > the same payload as say image.upload would likely create TONS > of > > > traffic. > > > > > Personally, I'm thinking about a batch payload, with a bare > > minimum of > > > the > > > > > following values: > > > > > > > > > > 'payload': [{'id': 'uuid1', 'owner': 'tenant1', 'created_at': > > > > > 'some_date', 'size': 100000000}, > > > > > {'id': 'uuid2', 'owner': 'tenant2', > 'created_at': > > > > > 'some_date', 'deleted_at': 'some_other_date', 'size': > 200000000}] > > > > > > > > > > That way the audit job/task could be configured to emit in > batches > > > which > > > > > a deployer could tweak the settings so as to not emit too many > > > messages. > > > > > I definitely welcome other ideas as well. > > > > > > > > Would it be better to group by tenant vs. image? > > > > > > > > One .exists per tenant that contains all the images owned by > > that tenant? > > > > > > > > -S > > > > > > > > > > > > > Thanks, > > > > > Andrew Melton > > > > > > > > > > > > > > > On Tue, Aug 13, 2013 at 4:27 AM, Julien Danjou > > <[email protected] <mailto:[email protected]> > > > > > <mailto:[email protected] <mailto:[email protected]>>> > wrote: > > > > > > > > > > On Mon, Aug 12 2013, Andrew Melton wrote: > > > > > > > > > > > So, my question to the Ceilometer community is this, > > does this > > > > > sound like > > > > > > something Ceilometer would find value in and use? If so, > > would > > > this be > > > > > > something > > > > > > we would want most deployers turning on? > > > > > > > > > > Yes. I think we would definitely be happy to have the > > ability to > > > drop > > > > > our pollster at some time. > > > > > I'm just concerned with the type of notification you'd > > send. It > > > has to > > > > > be enough fine grained so we don't lose too much > information. > > > > > > > > > > -- > > > > > Julien Danjou > > > > > // Free Software hacker / freelance consultant > > > > > // http://julien.danjou.info > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > OpenStack-dev mailing list > > > > > [email protected] > > <mailto:[email protected]> > > > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > > > _______________________________________________ > > > > OpenStack-dev mailing list > > > > [email protected] > > <mailto:[email protected]> > > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > _______________________________________________ > > > OpenStack-dev mailing list > > > [email protected] > > <mailto:[email protected]> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > [email protected] > > <mailto:[email protected]> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > [email protected] > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
