Hey,
Looks like your paste got a bit weird, I see thread 3 twice in the second
one? What were the exact commadns you did for each dump?
Is there any chance you could run the "memcached-debug" binary from the
build tree and get another stack dump? It might help, though this might be
enough information already.
Another interesting thing would be a stack dump, then 'continue' for a
bit, then stack dump again. See if it's making progress at all, or if the
hung thread moved around.
I see a bunch of sutff hanging on what's likely the central hash lock, and
then one thread sitting in assoc_find(), except so far as I know
assoc_find() itself can't block.
It's not clear to me if the other threads are waiting on the item lock
table or the global lock, so it'd be nice to find that out.
One thing you can try is setting the "-o hashpower=" value on start. Take
one of your servers that's been running a while and get the hash power
level from it, and use that to seed the new instance.
The new hash table expansion code is a bit tricky and could break like
this. So if you take one instance and pre-size the hash table, and that
stops it from hanging every day, please let us know. That'll help narrow
it down.
On Mon, 18 Feb 2013, lam wrote:
> the same again, right now:
>
> Thread 10 (Thread 0x7fbeab1c1700 (LWP 15511)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000004130a8 in item_get ()
> #2 0x0000000000409795 in process_command ()
> #3 0x000000000040b61a in drive_machine ()
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #5 0x00000000004129ad in worker_libevent ()
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #8 0x0000000000000000 in ?? ()
> Thread 9 (Thread 0x7fbeaa9c0700 (LWP 15512)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000004130a8 in item_get ()
> #2 0x0000000000409b56 in process_command ()
> #3 0x000000000040b61a in drive_machine ()
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #5 0x00000000004129ad in worker_libevent ()
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #8 0x0000000000000000 in ?? ()
> Thread 8 (Thread 0x7fbeaa1bf700 (LWP 15513)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000004130a8 in item_get ()
> #2 0x0000000000409b56 in process_command ()
> #3 0x000000000040b61a in drive_machine ()
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #5 0x00000000004129ad in worker_libevent ()
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #8 0x0000000000000000 in ?? ()
> Thread 7 (Thread 0x7fbea99be700 (LWP 15514)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000004130a8 in item_get ()
> #2 0x0000000000409795 in process_command ()
> #3 0x000000000040b61a in drive_machine ()
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #5 0x00000000004129ad in worker_libevent ()
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #8 0x0000000000000000 in ?? ()
> Thread 6 (Thread 0x7fbea91bd700 (LWP 15515)):
> #0 0x00007fbeabaf2353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007fbeac203883 in ?? () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #2 0x00007fbeac1ef450 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #3 0x00000000004129ad in worker_libevent ()
> #4 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
> Thread 5 (Thread 0x7fbea89bc700 (LWP 15516)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000004130a8 in item_get ()
> #2 0x0000000000409795 in process_command ()
> #3 0x000000000040b61a in drive_machine ()
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #5 0x00000000004129ad in worker_libevent ()
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #8 0x0000000000000000 in ?? ()
> Thread 4 (Thread 0x7fbea81bb700 (LWP 15517)):
> #0 0x00007fbeabaf2353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007fbeac203883 in ?? () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #2 0x00007fbeac1ef450 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #3 0x00000000004129ad in worker_libevent ()
> #4 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
> Thread 3 (Thread 0x7fbea79ba700 (LWP 15518)):
> #0 0x000000000041269d in assoc_find ()
> #1 0x0000000000411e52 in do_item_get ()
> #2 0x00000000004130b9 in item_get ()
> #3 0x0000000000409795 in process_command ()
> #4 0x000000000040b61a in drive_machine ()
> #5 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #6 0x00000000004129ad in worker_libevent ()
> #7 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #8 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #9 0x0000000000000000 in ?? ()
> Thread 2 (Thread 0x7fbea71b9700 (LWP 15519)):
> #0 0x00007fbeabdc8d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x0000000000412444 in assoc_maintenance_thread ()
> #2 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #3 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #4 0x0000000000000000 in ?? ()
> Thread 1 (Thread 0x7fbeac63d700 (LWP 15510)):
> #0 0x00007fbeabaf2353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007fbeac203883 in ?? () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #2 0x00007fbeac1ef450 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #3 0x0000000000404278 in main ()
>
>
> Thread 10 (Thread 0x7fbeab1c1700 (LWP 15511)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x00000000004130a8 in item_get ()
> No symbol table info available.
> #2 0x0000000000409795 in process_command ()
> No symbol table info available.
> #3 0x000000000040b61a in drive_machine ()
> No symbol table info available.
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #5 0x00000000004129ad in worker_libevent ()
> No symbol table info available.
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #8 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 9 (Thread 0x7fbeaa9c0700 (LWP 15512)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x00000000004130a8 in item_get ()
> No symbol table info available.
> #2 0x0000000000409b56 in process_command ()
> No symbol table info available.
> #3 0x000000000040b61a in drive_machine ()
> No symbol table info available.
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #5 0x00000000004129ad in worker_libevent ()
> No symbol table info available.
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #8 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 8 (Thread 0x7fbeaa1bf700 (LWP 15513)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x00000000004130a8 in item_get ()
> No symbol table info available.
> #2 0x0000000000409b56 in process_command ()
> No symbol table info available.
> #3 0x000000000040b61a in drive_machine ()
> No symbol table info available.
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #5 0x00000000004129ad in worker_libevent ()
> No symbol table info available.
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #8 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 7 (Thread 0x7fbea99be700 (LWP 15514)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x00000000004130a8 in item_get ()
> No symbol table info available.
> #2 0x0000000000409795 in process_command ()
> No symbol table info available.
> #3 0x000000000040b61a in drive_machine ()
> No symbol table info available.
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #5 0x00000000004129ad in worker_libevent ()
> No symbol table info available.
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #8 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 6 (Thread 0x7fbea91bd700 (LWP 15515)):
> #0 0x00007fbeabaf2353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #1 0x00007fbeac203883 in ?? () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #2 0x00007fbeac1ef450 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #3 0x00000000004129ad in worker_libevent ()
> No symbol table info available.
> #4 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #5 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #6 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 5 (Thread 0x7fbea89bc700 (LWP 15516)):
> #0 0x00007fbeabdc718b in pthread_mutex_trylock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x00000000004130a8 in item_get ()
> No symbol table info available.
> #2 0x0000000000409795 in process_command ()
> No symbol table info available.
> #3 0x000000000040b61a in drive_machine ()
> No symbol table info available.
> #4 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #5 0x00000000004129ad in worker_libevent ()
> No symbol table info available.
> #6 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #7 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #8 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 4 (Thread 0x7fbea81bb700 (LWP 15517)):
> #0 0x00007fbeabaf2353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #1 0x00007fbeac203883 in ?? () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #2 0x00007fbeac1ef450 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #3 0x00000000004129ad in worker_libevent ()
> No symbol table info available.
> #4 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #5 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #6 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 3 (Thread 0x7fbea79ba700 (LWP 15518)):
> #0 0x000000000041269d in assoc_find ()
> No symbol table info available.
> #1 0x0000000000411e52 in do_item_get ()
> No symbol table info available.
> #2 0x00000000004130b9 in item_get ()
> No symbol table info available.
> #3 0x0000000000409795 in process_command ()
> No symbol table info available.
> #4 0x000000000040b61a in drive_machine ()
> No symbol table info available.
> #5 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #6 0x00000000004129ad in worker_libevent ()
> #5 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #6 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 3 (Thread 0x7fbea79ba700 (LWP 15518)):
> #0 0x000000000041269d in assoc_find ()
> No symbol table info available.
> #1 0x0000000000411e52 in do_item_get ()
> No symbol table info available.
> #2 0x00000000004130b9 in item_get ()
> No symbol table info available.
> #3 0x0000000000409795 in process_command ()
> No symbol table info available.
> #4 0x000000000040b61a in drive_machine ()
> No symbol table info available.
> #5 0x00007fbeac1ef744 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #6 0x00000000004129ad in worker_libevent ()
> ---Type <return> to continue, or q <return> to quit---
> No symbol table info available.
> #7 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #8 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #9 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 2 (Thread 0x7fbea71b9700 (LWP 15519)):
> #0 0x00007fbeabdc8d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #1 0x0000000000412444 in assoc_maintenance_thread ()
> No symbol table info available.
> #2 0x00007fbeabdc4e9a in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #3 0x00007fbeabaf1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #4 0x0000000000000000 in ?? ()
> No symbol table info available.
> Thread 1 (Thread 0x7fbeac63d700 (LWP 15510)):
> #0 0x00007fbeabaf2353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #1 0x00007fbeac203883 in ?? () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #2 0x00007fbeac1ef450 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> No symbol table info available.
> #3 0x0000000000404278 in main ()
> No symbol table info available.
>
> strace:
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 7) = 0
> epoll_wait(3, {}, 32, 3) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
> epoll_wait(3, {}, 32, 10) = 0
>
> nobody 15510 123 7.4 2856392 2458320 ? Rl Feb13 10223:53
> /usr/bin/memcached -m 30720 -u nobody -t 8 -c 102400 -b 4096
>
> ls -l /proc/$(pidof memcached)/fd | wc -l
> 102401
>
> netstat -tpn|grep ':11211'|wc -l
> 11
>
> netstat -tpn|grep TIME_WAIT|wc -l
> 0
>
> top - 00:11:29 up 28 days, 7:01, 2 users, load average: 5.76, 4.80, 5.04
> Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 3.3 us, 4.1 sy, 0.0 ni, 86.7 id, 1.5 wa, 0.0 hi, 4.4 si, 0.0
> st
> KiB Mem: 32922088 total, 10381668 used, 22540420 free, 164520 buffers
> KiB Swap: 7812092 total, 9060 used, 7803032 free, 6289328 cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 15510 nobody 20 0 2789m 2.3g 1036 S 694.5 7.5 10240:02 memcached
>
> dmesg & syslog - clean
>
> memcached stats from an other memcached_node:
>
> array (
> 'pid' => '27721',
> 'uptime' => '496681',
> 'time' => '1361232187',
> 'version' => '1.4.15',
> 'libevent' => '2.0.19-stable',
> 'pointer_size' => '64',
> 'rusage_user' => '152865.721510',
> 'rusage_system' => '331487.768690',
> 'curr_connections' => '12880',
> 'total_connections' => '2227972771',
> 'connection_structures' => '14983',
> 'reserved_fds' => '40',
> 'cmd_get' => '19662471597',
> 'cmd_set' => '357943685',
> 'cmd_flush' => '0',
> 'cmd_touch' => '0',
> 'get_hits' => '15716410349',
> 'get_misses' => '3946061248',
> 'delete_misses' => '0',
> 'delete_hits' => '0',
> 'incr_misses' => '14592095',
> 'incr_hits' => '421800482',
> 'decr_misses' => '0',
> 'decr_hits' => '0',
> 'cas_misses' => '0',
> 'cas_hits' => '0',
> 'cas_badval' => '0',
> 'touch_hits' => '0',
> 'touch_misses' => '0',
> 'auth_cmds' => '0',
> 'auth_errors' => '0',
> 'bytes_read' => '783272430400',
> 'bytes_written' => '677759595016',
> 'limit_maxbytes' => '32212254720',
> 'accepting_conns' => '1',
> 'listen_disabled_num' => '0',
> 'threads' => '8',
> 'conn_yields' => '0',
> 'hash_power_level' => '22',
> 'hash_bytes' => '33554432',
> 'hash_is_expanding' => '0',
> 'bytes' => '1918954114',
> 'curr_items' => '6016624',
> 'total_items' => '779743733',
> 'expired_unfetched' => '293340687',
> 'evicted_unfetched' => '0',
> 'evictions' => '0',
> 'reclaimed' => '308439037',
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
