Hi Dormando, It took some time but we installed .17 and still observe the same issue with memory:
after 24 hours of run it consumed almost all memory from 48Gb available on dedicated box memcached process shows: Rss 43499344 Shared 540 Private 43498804 Swap 0 Pss 43498845 ================= with limit set to 24Gb: memcached -d -p 11211-m 24576 -c 128000 -P /var/run/memcached/memcached.pid -d -r -t 8 -o slab_reassign stats are: stats STAT pid 9023 STAT uptime 133857 STAT time 1390591888 STAT version 1.4.17 STAT libevent 1.4.13-stable STAT pointer_size 64 STAT rusage_user 153837.208189 STAT rusage_system 378991.192581 STAT curr_connections 6774 STAT total_connections 1155527288 STAT connection_structures 42638 STAT reserved_fds 40 STAT cmd_get 17749198141 STAT cmd_set 279314075 STAT cmd_flush 0 STAT cmd_touch 0 STAT get_hits 17065742324 STAT get_misses 683455817 STAT delete_misses 7573002573 STAT delete_hits 121708418 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 1 STAT cas_hits 5887 STAT cas_badval 5 STAT touch_hits 0 STAT touch_misses 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 2295246013025 STAT bytes_written 27303765419361 STAT limit_maxbytes 25769803776 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT threads 8 STAT conn_yields 0 STAT hash_power_level 25 STAT hash_bytes 268435456 STAT hash_is_expanding 0 STAT slab_reassign_running 0 STAT slabs_moved 0 STAT malloc_fails 0 STAT bytes 22652935663 STAT curr_items 31910169 STAT total_items 279313363 STAT expired_unfetched 35817 STAT evicted_unfetched 40144725 STAT evictions 81628302 STAT reclaimed 46197 Any idea where (and how) we can look where memory goes? Thank you, Den On Monday, December 30, 2013 12:55:07 PM UTC-8, Dormando wrote: > > On Mon, 30 Dec 2013, Den Samo wrote: > > > Dormando, thank you for so prompt reply.So, my historical query does not > show anything bad with�curr_connections - typical saw without growth > > trend. So we update to 17 and do observations. Also, as for fixed bug: > do you mean this > > one:� > https://github.com/memcached/memcached/commit/b2734f8321230bd52e36df7f82a6b1d71532e496? > > > That was one of them, yes. There's also some new counters and general > bugfixes; it's best to get a clean view of it. > > > On Monday, December 30, 2013 11:32:10 AM UTC-8, Dormando wrote: > > > hi Dormando, > > > we use�version 1.4.15. So it would be first step to update. > Thank you! > > > > > > >Does your memory�usage continue to go up while the number of > connections is stable, or do� > > > >the number of connections slowly increase with memory usage?� > > > > > > I am doing history query now. But to be sure, do you mean > "curr_connections" to compare or "connection_structures" or > > "total_connections" (this one > > > has HUGE value, see below) > > > > curr_connections is the current number of connected clients. > > connection_structure is something I forgot about: If that number > goes > > wacky that's a memory leak (but there's memory within the > structures isn't > > totalled up anywhere yet). > > total_connections is ticked up every time a client connects. so a > > connect/disconnect/connect ups it by 2. > > > > I'm curious if your memory leaks as curr_connections grows, or if > that > > number is relatively static (going up and down with traffic > levels) but > > memory continually slowly increases. > > > > Please let me know when you've tried 1.4.17 - and if it still > leaks, > > provide 'stats' output from that version. Thanks! > > > > > here is the stats from one of the servers: > > > > > > stats > > > STAT pid 10409 > > > STAT uptime 322887 > > > STAT time 1388427859 > > > STAT version 1.4.15 > > > STAT libevent 1.4.13-stable > > > STAT pointer_size 64 > > > STAT rusage_user 230345.511159 > > > STAT rusage_system 354573.674608 > > > STAT curr_connections 4151 > > > STAT total_connections 1251550661 > > > STAT connection_structures 6655 > > > STAT reserved_fds 40 > > > STAT cmd_get 26733462674 > > > STAT cmd_set 291587557 > > > STAT cmd_flush 0 > > > STAT cmd_touch 0 > > > STAT get_hits 26157644553 > > > STAT get_misses 575818121 > > > STAT delete_misses 6433865116 > > > STAT delete_hits 91028651 > > > STAT incr_misses 0 > > > STAT incr_hits 0 > > > STAT decr_misses 0 > > > STAT decr_hits 0 > > > STAT cas_misses 0 > > > STAT cas_hits 28992 > > > STAT cas_badval 22 > > > STAT touch_hits 0 > > > STAT touch_misses 0 > > > STAT auth_cmds 0 > > > STAT auth_errors 0 > > > STAT bytes_read 7326099432348 > > > STAT bytes_written 31415635467012 > > > STAT limit_maxbytes 25769803776 > > > STAT accepting_conns 1 > > > STAT listen_disabled_num 0 > > > STAT threads 8 > > > STAT conn_yields 0 > > > STAT hash_power_level 25 > > > STAT hash_bytes 268435456 > > > STAT hash_is_expanding 0 > > > STAT slab_reassign_running 0 > > > STAT slabs_moved 0 > > > STAT bytes 22814358539 > > > STAT curr_items 34689744 > > > STAT total_items 291584654 > > > STAT expired_unfetched 27821 > > > STAT evicted_unfetched 91912345 > > > STAT evictions 135352494 > > > STAT reclaimed 34495 > > > > > > -Den > > > > > > On Friday, December 27, 2013 5:14:16 PM UTC-8, Dormando wrote: > > > � � � > Hi everybody,What would be best practice to set > max connections? We have "memory leak" issue (memory usage grows, eats all > > available > > > � � � memory, > > > � � � > process crashes). I use quotes as I am not sure > that this is actually memory leak. I still thinking on the way to validate > > this. One > > > � � � potential > > > � � � > suspect is memory used for network connections. We > have -c 128000. And -m 24576 as memory limiter. the box itself has 48G of > > memory > > > � � � and dedicated > > > � � � > to memcache. > > > � � � > > > > � � � > Also, more abstract question: is there any way to > see how memcache uses memory? like what amount was reserved for > > connections? > > > � � � > > > > � � � > Thank you and Happy Holidays, > > > � � � > Den > > > > > > � � � Hi! > > > > > > � � � What version are you running? If it's not 1.4.17, > can you try that version > > > � � � and see if it still "leaks"? There was at least one > bug fix in the last > > > � � � two versions which could drop a connection > structure. > > > > > > � � � The "stats" output can tell you how much memory is > dictated for the slab > > > � � � cache and for the hash table, but not all types of > memory are accounted > > > � � � presently. > > > > > > � � � That is a very large number of connections though. > Can you provide some > > > � � � 'stats' output? How many connections do you usually > have? Does your memory > > > � � � usage continue to go up while the number of > connections is stable, or do > > > � � � the number of connections slowly increase with > memory usage? > > > > > > -- > > > � > > > --- > > > You received this message because you are subscribed to the > Google Groups "memcached" group. > > > To unsubscribe from this group and stop receiving emails from > it, send an email to [email protected]. > > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > > > > > -- > > � > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
