Hey, I can add some more accounting this weekend and make a branch for you to try. Do you have a better process for quickly iterating from now on? It's going to suck if it's weeks between versions.
If the accounting doesn't catch it or make it obvious we might have to take a core dump of a leaked process and examine it. On Fri, 24 Jan 2014, Den Samo wrote: > Hi Dormando, > > It took some time but we installed .17 and still observe the same issue with > memory: > > after 24 hours of run it consumed almost all memory from 48Gb available on > dedicated box memcached process shows: > > Rss 43499344 > Shared 540 > Private 43498804 > Swap 0 > Pss 43498845 > ================= > > with limit set to 24Gb: > memcached -d -p 11211-m 24576 -c 128000 -P /var/run/memcached/memcached.pid > -d -r -t 8 -o slab_reassign > > stats are: > stats > STAT pid 9023 > STAT uptime 133857 > STAT time 1390591888 > STAT version 1.4.17 > STAT libevent 1.4.13-stable > STAT pointer_size 64 > STAT rusage_user 153837.208189 > STAT rusage_system 378991.192581 > STAT curr_connections 6774 > STAT total_connections 1155527288 > STAT connection_structures 42638 > STAT reserved_fds 40 > STAT cmd_get 17749198141 > STAT cmd_set 279314075 > STAT cmd_flush 0 > STAT cmd_touch 0 > STAT get_hits 17065742324 > STAT get_misses 683455817 > STAT delete_misses 7573002573 > STAT delete_hits 121708418 > STAT incr_misses 0 > STAT incr_hits 0 > STAT decr_misses 0 > STAT decr_hits 0 > STAT cas_misses 1 > STAT cas_hits 5887 > STAT cas_badval 5 > STAT touch_hits 0 > STAT touch_misses 0 > STAT auth_cmds 0 > STAT auth_errors 0 > STAT bytes_read 2295246013025 > STAT bytes_written 27303765419361 > STAT limit_maxbytes 25769803776 > STAT accepting_conns 1 > STAT listen_disabled_num 0 > STAT threads 8 > STAT conn_yields 0 > STAT hash_power_level 25 > STAT hash_bytes 268435456 > STAT hash_is_expanding 0 > STAT slab_reassign_running 0 > STAT slabs_moved 0 > STAT malloc_fails 0 > STAT bytes 22652935663 > STAT curr_items 31910169 > STAT total_items 279313363 > STAT expired_unfetched 35817 > STAT evicted_unfetched 40144725 > STAT evictions 81628302 > STAT reclaimed 46197 > > Any idea where (and how) we can look where memory goes? > > Thank you, > Den > > On Monday, December 30, 2013 12:55:07 PM UTC-8, Dormando wrote: > On Mon, 30 Dec 2013, Den Samo wrote: > > > Dormando, thank you for so prompt reply.So, my historical query does > not show anything bad with�curr_connections - typical saw > without growth > > trend. So we update to 17 and do observations. Also, as for fixed > bug: do you mean this > > > one:�https://github.com/memcached/memcached/commit/b2734f8321230bd52e36df7f82a6b1d71532e496 > ? > > That was one of them, yes. There's also some new counters and general > bugfixes; it's best to get a clean view of it. > > > On Monday, December 30, 2013 11:32:10 AM UTC-8, Dormando wrote: > > > hi Dormando, > > > we use�version 1.4.15. So it would be first step to update. > Thank you! > > > > > > >Does your memory�usage continue to go up while the number > of connections is stable, or do� > > > >the number of connections slowly increase with memory > usage?� > > > > > > I am doing history query now. But to be sure, do you mean > "curr_connections" to compare or "connection_structures" or > > "total_connections" (this one > > > has HUGE value, see below) > > > > curr_connections is the current number of connected clients. > > connection_structure is something I forgot about: If that > number goes > > wacky that's a memory leak (but there's memory within the > structures isn't > > totalled up anywhere yet). > > total_connections is ticked up every time a client connects. so > a > > connect/disconnect/connect ups it by 2. > > > > I'm curious if your memory leaks as curr_connections grows, or > if that > > number is relatively static (going up and down with traffic > levels) but > > memory continually slowly increases. > > > > Please let me know when you've tried 1.4.17 - and if it still > leaks, > > provide 'stats' output from that version. Thanks! > > > > > here is the stats from one of the servers: > > > > > > stats > > > STAT pid 10409 > > > STAT uptime 322887 > > > STAT time 1388427859 > > > STAT version 1.4.15 > > > STAT libevent 1.4.13-stable > > > STAT pointer_size 64 > > > STAT rusage_user 230345.511159 > > > STAT rusage_system 354573.674608 > > > STAT curr_connections 4151 > > > STAT total_connections 1251550661 > > > STAT connection_structures 6655 > > > STAT reserved_fds 40 > > > STAT cmd_get 26733462674 > > > STAT cmd_set 291587557 > > > STAT cmd_flush 0 > > > STAT cmd_touch 0 > > > STAT get_hits 26157644553 > > > STAT get_misses 575818121 > > > STAT delete_misses 6433865116 > > > STAT delete_hits 91028651 > > > STAT incr_misses 0 > > > STAT incr_hits 0 > > > STAT decr_misses 0 > > > STAT decr_hits 0 > > > STAT cas_misses 0 > > > STAT cas_hits 28992 > > > STAT cas_badval 22 > > > STAT touch_hits 0 > > > STAT touch_misses 0 > > > STAT auth_cmds 0 > > > STAT auth_errors 0 > > > STAT bytes_read 7326099432348 > > > STAT bytes_written 31415635467012 > > > STAT limit_maxbytes 25769803776 > > > STAT accepting_conns 1 > > > STAT listen_disabled_num 0 > > > STAT threads 8 > > > STAT conn_yields 0 > > > STAT hash_power_level 25 > > > STAT hash_bytes 268435456 > > > STAT hash_is_expanding 0 > > > STAT slab_reassign_running 0 > > > STAT slabs_moved 0 > > > STAT bytes 22814358539 > > > STAT curr_items 34689744 > > > STAT total_items 291584654 > > > STAT expired_unfetched 27821 > > > STAT evicted_unfetched 91912345 > > > STAT evictions 135352494 > > > STAT reclaimed 34495 > > > > > > -Den > > > > > > On Friday, December 27, 2013 5:14:16 PM UTC-8, Dormando wrote: > > > � � � > Hi everybody,What would be best practice to set > max connections? We have "memory leak" issue (memory usage > grows, eats all > > available > > > � � � memory, > > > � � � > process crashes). I use quotes as I am not sure > that this is actually memory leak. I still thinking on the way > to validate > > this. One > > > � � � potential > > > � � � > suspect is memory used for network connections. > We have -c 128000. And -m 24576 as memory limiter. the box > itself has 48G of > > memory > > > � � � and dedicated > > > � � � > to memcache. > > > � � � > > > > � � � > Also, more abstract question: is there any way > to see how memcache uses memory? like what amount was reserved > for > > connections? > > > � � � > > > > � � � > Thank you and Happy Holidays, > > > � � � > Den > > > > > > � � � Hi! > > > > > > � � � What version are you running? If it's not 1.4.17, > can you try that version > > > � � � and see if it still "leaks"? There was at least > one bug fix in the last > > > � � � two versions which could drop a connection > structure. > > > > > > � � � The "stats" output can tell you how much memory > is dictated for the slab > > > � � � cache and for the hash table, but not all types > of memory are accounted > > > � � � presently. > > > > > > � � � That is a very large number of connections > though. Can you provide some > > > � � � 'stats' output? How many connections do you > usually have? Does your memory > > > � � � usage continue to go up while the number of > connections is stable, or do > > > � � � the number of connections slowly increase with > memory usage? > > > > > > -- > > > � > > > --- > > > You received this message because you are subscribed to the > Google Groups "memcached" group. > > > To unsubscribe from this group and stop receiving emails from > it, send an email to [email protected]. > > > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > > > > > -- > > � > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, > send an email to [email protected]. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
