Hi, Thanks for the response, i believe it won't work. First i have many items of different sizes, so i can't get them to one slab, having 2 clients would be a performance hit because i'd need to make connections twice every request (unfortunately the part that handles SQL write stagging is reading/writing some cache data too)
>But pulling the full list of keys back so you can find and fast reclaim some >is definitely the wrong way to do that Why it's wrong? Maybe we can pull the items inside memcached and then call item_lock=>do_item_unlink=>do_item_remove=>item_unlock If we managed to fork the client every minute we won't have to use locking and it probably won't use too much memory too. If it won't be possible maybe we could lock one slab (not whole slab class) and just process a copy so we can have some inexpensive garbage collection. LRU is very lightweight (yes i get the point of not having long lasting LOCKs) but not suitable for all use patterns, because it can miss keys then you have uncontrolled slab growth and then you need to restart (well from my experience it does most of the time when you start to have different expiration times). Eg. last time (before using this PULL_LIST->CALL_GET method i had 20 million keys in cache, and all slabs in 3 classes now i have about 20-30k). Maybe we could even get rid of all the code that does garbage collection and LRU if we just make LRU to be on-slab-cleanup instead of on-item-access? (eg. every 10 seconds we'd process all items from LRU slab from top to bottom, then move the slab to bottom of LRU list). This maybe would allow to have just one-way list instead of 2 way list and play better with CPU cache, so be more memory efficient. I have read the code for about 1 hour or two so probably most of my assumptions are wrong. Can anyone suggest most efficient way of making fast reclaim? For now im thinking about forking to be honest... i know 2GB lost is not much memory but with current LRU approach i can't see any metrics on how much memory i need to keep the thing using memcached running and even if i use only about 30-40MB of those 2GB for not-expired items i still get evictions. If we have this kind of garbage collection would you include this into the client? Don't know if I'll be able to do this or if there's anyone willing to help with this? Easiest thing would be probably just increase the LRU keys scanned to eg. 10k but it'll probably just kill the server :) Thanks, Slawomir. On 26 Lut, 10:36, [email protected] wrote: > Updates: > Status: Invalid > > Comment #1 on issue 256 by [email protected]: PLEASE do not remove > cachedump, better dumping feature > neededhttp://code.google.com/p/memcached/issues/detail?id=256 > > Why not just run two memcached instances? One for short expiration times, > one for longer. > > Or as a grosser hack, pad items with longer expiration times so they occupy > a different slab. Since each slab has its own LRU, you'd trade a few bytes > of overhead (it's not much at the smaller sizes) for "fast" reclaim. > > But pulling the full list of keys back so you can find and fast reclaim > some is definitely the wrong way to do that. I tried to start a thread on > the mailing list a while ago for ideas on stats cachedump but nobody really > responded to it. In most cases what people want out of it isn't the best > usage of resources, and yours seems to be along those lines as well. > > *but*, having said that, I have given thought recently to improving the > efficiency given your example traffic pattern. Given a few months time > memcached will probably just handle your situation better out of the box, > and you can use one of the above workarounds for now without having to > change core code and rely on your own fork in production. > > For other people reading this issue; this is the sort of thing for a thread > on the mailing list. This isn't a bug report. If it were more focused on "I > have this traffic pattern and it's not very efficient" that would be more > on target. > > If you wish to discuss this further please hit up the mailing list or open > a bug report on the real issue. > > thanks!
