On 08/08/2013 11:36 PM, Angus Salkeld wrote: > On 08/08/13 13:16 -0700, Clint Byrum wrote: >> Last night while reviewing a feature which would add more events to the >> event table, it dawned on me that the event table really must be removed. > > >> >> https://bugs.launchpad.net/heat/+bug/1209492 >> >> tl;dr: users can write an infinite number of rows to the event table at >> a fairly alarming rate just by creating and updating a very large stack >> that has no resources that cost any time or are even billable (like an >> autoscaling launch configuration). >> >> The table has no purge function, so the only way to clear out old events >> is to delete the stack, or manually remove them directly in the database. >> >> We've all been through this before, logging to a database seems great >> until you actually do it. >> >> I have some ideas for how to solve it, but I wanted to get a wider >> audience: >> >> 1) Make the event list a ring buffer. Have rows 0 - $MAX_BUFFER_SIZE in >> each stack, and simply write each new event to the next open position, >> wrapping at $MAX_BUFFER_SIZE. Pros: little change to current code, >> just need an offset column added and code that will properly wrap to 0 >> at $MAX_BUFFER_SIZE. Cons: still can incur heavy transactional load on >> the database server.A >> >> 1.b) Same, but instead of rows, just maintain a blob and append the rows >> as json list. Lowers transactional load but would push some load onto >> the API servers and such to parse these out, and would make pagination >> challenging. Blobs also can be a drain on DB server performance. >> >> 2) Write a purge script. Delete old ones. Pros: No code change, just >> new code to do purging. Cons: same as 1, plus more vulnerability to an >> aggressive attacker who can fit a lot of data in between purges. Also >> large scale deletes can be really painful (see: keystone sql token >> backend). >> >> 3) Log events to Swift. I can't seem to find information on how/if >> appending works there. Tons of tiny single-row files is an option, but I >> want to hear from people with more swift knowledge if that is a viable, >> performant option. Pros: Scale to the moon. Can charge tenant for usage >> and let them purge events as needed. Cons: Adds swift as a requirement >> of Heat. >> >> 4) Provide a way for users to receive logs via HTTP POST. Pros: Simple >> and punts the problem to the users. Cons: users will be SoL if they >> don't have a place to have logs posted to. >> >> 5) Provide a way for users to receive logs via messaging service like >> Marconi. Pros/Cons: same as HTTP, but perhaps a little more confusing >> and ambitious given Marconi's short existence. >> >> 6) Provide a pluggable backend for logging. This seems like the way most >> OpenStack projects solve these issues, which is to let the deployers >> choose and/or provide their own way to handle a sticky problem. Pros: >> Simple and flexible for the future. Cons: Would require writing at least >> one backend provider that does what the previous 5 options suggest. >> >> To be clear: Heat cannot really exist without this, as it is the only way >> to find out what your stack is doing or has done. > > btw Clint I have ditched that "Recorder" patch as Ceilometer is > getting a Alarm History api soon, so we can defer to that for that > functionality (alarm transitions). > > But we still need a better way to record events/logs for the user. > So I make this blueprint a while ago: > https://blueprints.launchpad.net/heat/+spec/user-visible-logs > > I am becomming more in favor of user options rather than deployer > options if possible. So provide resources for Marconi, Meniscus and > what ever... > Although what is nice about Marconi is you could then hook up what > ever you want to it.
Logs are one thing (and Meniscus is a great choice for that), but events are the very thing CM is designed to handle. Wouldn't it make sense to push them back into there? > -Angus > > >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev