Re: Idea for reclaimation algo

2014-04-13 Thread dormando
Yes :) If I recall how that works, it's mildly similar to a few other things I've seen. Not super trivial to implement in a short period of time though. On Sat, 12 Apr 2014, Ryan McElroy wrote: Facebook implemented a visitor plugin system that we use to kick out already-expired items in our

Re: Idea for reclaimation algo

2014-04-13 Thread dormando
Hey Dormando... Some quick question first... i have checked some Intel papers on their memcached fork and for 1.6 it seems that there's some rather big lock contention... have you thought about just gluing individual items to a thread, using maybe item hash or some configurable method...

Re: Idea for reclaimation algo

2014-04-13 Thread dormando
On Sun, 13 Apr 2014, Slawomir Pryczek wrote: So high evictions when cleaning algo isn't enabled could be caused by slab imbalance due to high-memory slabs eating most of ram... and i just incorrectly assumed low TTL items are expired before high TTL items, because in such cases the

Re: Idea for reclaimation algo

2014-04-11 Thread Slawomir Pryczek
Hi Dormando, more about the behaviour... when we're using normal memcached 1.4.13 16GB of memory gets exhausted in ~1h, then we start to have almost instant evictions of needed items (again these items aren't really needed individually, just when many of them gets evicted it's unacceptable

Re: Idea for reclaimation algo

2014-04-11 Thread Les Mikesell
On Fri, Apr 11, 2014 at 3:47 AM, Slawomir Pryczek slawek1...@gmail.com wrote: When using memcached as a buffer for mysql writes, we know exactly what to hit and when. Short TTL expired items, pile up near the head... long TTL live items pile up near the tail and it's creating a barrier that

Re: Idea for reclaimation algo

2014-04-11 Thread dormando
On Fri, 11 Apr 2014, Slawomir Pryczek wrote: Hi Dormando, more about the behaviour... when we're using normal memcached 1.4.13 16GB of memory gets exhausted in ~1h, then we start to have almost instant evictions of needed items (again these items aren't really needed individually, just

Re: Idea for reclaimation algo

2014-04-11 Thread dormando
s/pagging/padding/. gah. On Fri, 11 Apr 2014, dormando wrote: On Fri, 11 Apr 2014, Slawomir Pryczek wrote: Hi Dormando, more about the behaviour... when we're using normal memcached 1.4.13 16GB of memory gets exhausted in ~1h, then we start to have almost instant evictions of needed

Re: Idea for reclaimation algo

2014-04-10 Thread Slawomir Pryczek
Hi Dormando, thanks for suggestions, background thread would be nice... The idea is actually that with 2-3GB i get plenty of evictions of items that need to be fetched later. And with 16GB i still get evictions, actually probably i could throw more memory than 16G and it'd only result in more

Re: Idea for reclaimation algo

2014-04-10 Thread Guille -bisho-
My recommendation: Create two memcache instances listening in different ports, one for important, need-to-be-persistent data and other for best effort. Find the proper size of the important instance so all your working dataset fits in, and the rest of the ram in the box for the best effort

Re: Idea for reclaimation algo

2014-04-10 Thread Brian Moon
+1 to everything dormando just said. Brian. http://brian.moonspot.net/ On 4/10/14, 13:53 , dormando wrote: You really really really really really *must* not put data in memcached which you can't lose. Seriously, really don't do it. If you need persistence, try using a redis instance

Re: Idea for reclaimation algo

2014-04-10 Thread Slawomir Pryczek
Hey Dormando, thanks again for some comments... appreciate the help. Maybe i wasn't clear enough. I need only 1 minute persistence, and i can lose data sometimes, just i can't keep loosing data every minute due to constant evictions caused by LRU. Actually i have just wrote that in my previous

Re: Idea for reclaimation algo

2014-04-10 Thread dormando
Hey Dormando, thanks again for some comments... appreciate the help. Maybe i wasn't clear enough. I need only 1 minute persistence, and i can lose data sometimes, just i can't keep loosing data every minute due to constant evictions caused by LRU. Actually i have just wrote that in my

Idea for reclaimation algo

2014-04-09 Thread Slawomir Pryczek
Hi Guys, im running a specific case where i don't want (actually can't have) to have evicted items (evictions = 0 ideally)... now i have created some simple algo that lock the cache, goes through linked list and evicts items... it makes some problems, like 10-20ms cache locks on some cases.

Re: Idea for reclaimation algo

2014-04-09 Thread dormando
Hi Guys, im running a specific case where i don't want (actually can't have) to have evicted items (evictions = 0 ideally)... now i have created some simple algo that lock the cache, goes through linked list and evicts items... it makes some problems, like 10-20ms cache locks on some cases.