Hi Dormando, thanx for this great explanation!
On Sat, Mar 13, 2010 at 2:07 AM, dormando <[email protected]> wrote: > > - On read, if a key is past its expiry time, return its memory to the slab > pool and return NOT FOUND > - On write, try to allocate new memory: > * From the slab's pool of free memory. > * ... and if the slab is full, add another megabyte from the main pool > * ... if the main pool is fully assigned out to slab classes, find > something to evict. > * for that slab class, look at the LRU tail (oldest items in the cache) > * walk up the tail looking for *items that are already expired* > * after 50 items have been examined, if no expired items are found, > Cool. Would it be possible to make this number configurable via a cmd line switch? > * walk the list again, and *expire* the first thing you find > (I'm leaving out refcount locks for sake of this discussion) > [...] > This does mean, by way of painfully describing how an LRU works, that the > odds of you finding sessions in memcached which have not been expired, but > are being evicted from the LRU earlier than expired sessions, is very > unlikely. > That's right for the normal case. However, for the memcached-session-manager I just implemented a feature so that sessions are only sent to memcached for backup if session data was modified. To prevent expiration of the session in memcached, a background thread is updating sessions in memcached that would expire ~20 seconds later in memcached (if there would be some kind of "touch" operation I'd just use that one). When they are updated, they are set with an expiration of the remaining expiration time in tomcat. This would introduce the issue, that the update would push them to the head of the LRU, but their expiration might be e.g. only 5 minutes. So they would be expired but won't be reached when the 50 items are checked for expiration. [...] > So, lets say your traffic ends up looking like: > > - For the first 10,000 sessions, they are all 200 kilobytes. This ends up > having memcached allocate all of its slab memory toward something that > will fit 200k items. > - You get linked from the frontpage of digg.com and suddenly you have a > bunch of n00bass users hitting your site. They have smaller sessions since > they are newbies. 10k items. > - Memcached has only reserved 1 megabyte toward 10k items. So now all of > your newbies share a 1 megabyte store for sessions, instead of 200 > megabytes. > > If you set the minimum slab size to 200k, you unify the memory so the > largest pool of memory is always available for your users. However, you > drastically raise the memory overhead for users with 10k sessions. 90%+ > overhead. That's true, this is the drawback that I'd have to accept. Do you see other disadvantages with this approach (e.g. performance wise)? > In reality when you start a memcached instance and throw traffic > at it, sessions will either be very close together in size, or vary by > some usable distribution. > Yes, for this I (the memcached-session-manager) should provide some stats on size distribution. Or does memcached already provide this information via its stats? > If it ends up changing over time, the only present workaround is to > restart memcached. This sucks since it'll end up logging your users out. > In my case this is not that much an issue (users won't get logged out), as sessions are served from local memory. Session are only in memcached for the purpose of session failover. So this restart could be done when operations could be *sure* that no tomcat will die. [...] > However the slab out of balance thing is a real fault of ours. It's a > project on my plate to have automated slab rebalancing done in some usable > fashion within the next several weeks. This means that if a slab is out of > memory and under pressure, memcached will decide if it can pull memory > from another slab class to satisfy that need. As the size of your items > change over time, it will thus try to compensate. > Yeah, great! Cheers, Martin
