Aaron once mention working on having the keys distibuted over many servers. I think this would involve setting up some algorith to decide which remote server to look at. Key < 100 to server 1, Key > 100 to server2. Not sure how easy it would be to implement some generic way of handle various different types of keys.
I'd be very interested to see how far you can scale the disk cache. Regards Al On 7/19/06, Aaron Smuts <[EMAIL PROTECTED]> wrote:
--- "Schwarz, Peter" <[EMAIL PROTECTED]> wrote: > This depends on how your test is written. One thing > that we noticed was > that the cache loses writes of data (through the > purgatory), if you don't > wait long enough. We had, for 2 million items, each > 16k, a cache size of > 33GB, which was inline with the computed size for > the cache. > There is an optimization, where items are removed from the queue if they are plucked out of purgatory. This would cause problems for memory size = 0 configurations, since the item will be lost. I'm working on a fix since I run in a 0 memory configuration on my remote server cluster. > I'm not sure what MySQL would be doing to keep that > size down. > MySQL uses more space if anything. I'm just using it so I can have a background cleanup process. I had to build in processes to optimize myisam on a schedule. > As for the memory limitations, the key set is > actually not too bad, since > we're using pretty small keys (Integers for the > tests). With a 512M vm > size, this wasn't an issue. Very interesting. I'm mainly running multiple small < 128MB max apps on a box. . . . > > > The optimization routine is fairly crude. > > Using a database isn't an option for us (I know, it > sounds strange, but hey, > sometimes the powers that be make certain calls...), > so we're planning on > making changes to the defrag/recycle portions of the > IndexDiskCache. We > would like to submit the changes for contribution. > What would the process > be for that? I'd love to see improvements in this part of the indexed disk cache. Please send patches along with unit test so I can evaluate them. What exactly are you doing? Do you want to optimize in place using a larger recycle bin? If the changes are major, we can consider adding another disk cache implemention in addition to the ones that we currently support. Cheers, Aaron > > Cheers, > > Peter > > --------------------------------------------------------------------- > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
