> Hello All !))I have a some problem. > We have 3 memcached nodes cluster (memcached version: 1.4.5). > Average loading on every node is about 300-350 req/s. > We are interacting with memcached throw JAVA-API (net.spy.memcached) > But sometimes (very rarely) we gets error lines in our application log, as > folowing: > > 01:00:59 ERROR Error: Out of memory > 01:30:01 ERROR Error: Out of memory > 03:30:25 ERROR Error: Out of memory > 11:11:16 ERROR Error: Out of memory > > > I can't found any dependencies between these errors and software environment. > And anything: > I found "outofmemory" errors on some slabs (stats slabs and stats items > requests) as folowing: > > server 1: > > ........ > STAT items:39:number 1 > STAT items:39:age 27457197 > STAT items:39:evicted 3720 > STAT items:39:evicted_nonzero 3 > STAT items:39:evicted_time 10 > STAT items:39:outofmemory 41 > STAT items:39:tailrepairs 0 > STAT items:39:reclaimed 0 > .......... > > stats slabs > ... > STAT 39:chunk_size 493552 > STAT 39:chunks_per_page 2 > STAT 39:total_pages 1 > STAT 39:total_chunks 2 > STAT 39:used_chunks 1 > STAT 39:free_chunks 1 > STAT 39:free_chunks_end 0 > STAT 39:mem_requested 415638 > STAT 39:get_hits 562215 > STAT 39:cmd_set 568308 > STAT 39:delete_hits 0 > STAT 39:incr_hits 0 > STAT 39:decr_hits 0 > STAT 39:cas_hits 0 > STAT 39:cas_badval 0 > .... > STAT active_slabs 42 > STAT total_malloced 2151880632
You should upgrade to the latest version. It might mitigate this a bit better. The problem is most likely that this is a very high slab class. You cans ee it's big enough that it can only fit two items per page. It only has one page assigned to the slab class. So there can be only *two* items of that size stored in your memcached instance. Memcached can only store a new item if it has free memory to work with, or if it can evict a previous item. However, if you're in the process of fetching or storing an item, the item might be locked. If that happens it simply says "out of memory". You can cause this pretty easily by uploading 3 items in parallel. The first two will work but the third won't be able to get memory. New versions work a bit better, or you can enable slab rebalancing to allow more memory to be put into the higher slab class when necessary. Sorry. It's a bit of an obscure tradeoff. :/ -Dormando -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
