Antony T Curtis wrote: > On 11 Jul 2008, at 17:12, Brian Aker wrote: > >> Hi! >> >> Lets figure out a memory policy first. I talked to Jay about this for >> a bit today. What I would like to do is sit down at OSCON and write up >> a plan for how to tackle memory. >> >> We have (at a least): >> >> 1) Short lived, per query. >> 2) Medium lived, per login session >> 3) Forever >> >> We also have the issue that some things, like Tables, are not per >> session and are cached. > > Here is my $0.02... > > Table/view/schema object definitions should all be persistent (forever). > This makes it cheap to check if a table exists without having to hit the > disk. An async event notification system should exist so that a storage > engine can inform the server of the creation of a new table so that the > server can request and cache it.
Actually, this is precisely what Brian and I talked about a couple weeks ago, and that I have been prototyping locally. Essentially, a central cache of metadata has a thread pool of worker threads that respond to signals from either readers or updaters. The worker threads responding to update requests (from storage engines, the system itself (for non-SE-specfic stuff), or other plugins) uses an RCU strategy to avoid locking entirely. A spinlock is used around the critical struct replacement in the cache and then calls rcu_synchronize() to clean up the particular metadata resource. Readers call metadata_get_resource() which returns an rcu protected pointer to a MetadataResource, which can be used to determine table structure, schema information, etc... I'm certainly no-where-near complete on the prototype but I think the logic around the design is fairly good, but unfortunately, it doesn't solve the problem that Mark and Stewart bring up about engines that don't know when to notify the listeners in the central cache... :( Definitely something to chat about at OSCON. > It may be a good idea to keep open storage handlers on a per-session > basis and not shared. Then the problem of sharing/limiting open file > handles is pushed down into the storage engines. Of course, a ceiling > for max number of open handler instances per session is useful... Max of > 63 per connection is reasonable as that is max number of tables in a join. > Perhaps specify on a per-engine basis what is max number of file handles > that engine may open. Perhaps a small utility helper can be provided to > retrofit this facility like what I did with Amira to support 32000 > "open" file handles. > The current scheme mysqld uses which limits the total number of handler > instances doesn't maximize use of available file handles, especially > with many storage engines which are not file-per-table and share file > handles among multiple handler instances even if they are file-per-table. > > A small added bonus if you do this... some memory managers on some OS > have limited NUMA knowledge and will then try to allocate objects in > memory local or near to the current cpu. As we move to massively > parallel system, we will see more and more NUMA-like architectures. Of > course, the most common NUMA systems today are all those HyperChannel > AMD SMP systems. Having a shared global pool can hurt performance on them. > > Regards, > Antony > >> >> I would like to define these out, and create a strategy that is >> written down and someone and go to when they ask "how do I allocate >> memory?" >> And everything should be error checked/type checked. >> >> Until we have a plan though, lets just clean up what we have to make >> it a bit easier to deal with when we have a plan. >> >> Cheers, >> -Brian >> >> >> On Jul 11, 2008, at 6:36 AM, Stewart Smith wrote: >> >>> On Fri, 2008-07-11 at 12:28 +0100, David LP Travel wrote: >>>> Original 1990ties MySQL allocation always was per query on a stack. And >>>> when the query finished you reset the stack. >>>> >>>> Very fast and very simple. When we added things like stored procedures >>>> etc that way broke. It looks like drizzle could go back to the original >>>> way... >>> >>> /me cheers "talloc! talloc!" >>> >>> :) >>> -- >>> Stewart Smith ([EMAIL PROTECTED]) >>> http://www.flamingspork.com/ >>> >>> _______________________________________________ >>> Drizzle mailing list >>> [EMAIL PROTECTED] >>> http://lists.tangent.org/mailman/listinfo/drizzle >> >> -- >> _______________________________________________________ >> Brian "Krow" Aker, brian at tangent.org >> Seattle, Washington >> http://krow.net/ <-- Me >> http://tangent.org/ <-- Software >> _______________________________________________________ >> You can't grep a dead tree. >> >> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~drizzle-discuss >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~drizzle-discuss >> More help : https://help.launchpad.net/ListHelp > > > _______________________________________________ > Mailing list: https://launchpad.net/~drizzle-discuss > Post to : [email protected] > Unsubscribe : https://launchpad.net/~drizzle-discuss > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
