What version of memcached is this?
How are you reproducing the bug?
What OS/version are you on?
On Wed, 25 Apr 2012, lee wrote:
> the memcached main process can't accept connect when use
> assoc_maintenance_thread function,here is the gdb bt result:
> we have 4 thread :
> |-memcached,28059 -d -p 12111 -c 8192 -m 16384 -u root -l 127.0.0.1
> | |-{memcached},28060
> | |-{memcached},28061
> | |-{memcached},28062
> | |-{memcached},28063
> | `-{memcached},28064
>
> gdb bt result:
> gdb -p 28060
> GNU gdb Fedora (6.8-37.el5)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
> gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Attaching to process 28060
>
> warning: process 28060 is a cloned process
> Reading symbols from /usr/local/sinawap/bin/memcached...done.
> Reading symbols from /usr/local/sinawap/lib/libevent-1.4.so.2...done.
> Loaded symbols for /usr/local/sinawap/lib/libevent-1.4.so.2
> Reading symbols from /lib64/libpthread.so.0...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 0x7f2ec05b86e0 (LWP 28059)]
> [New Thread 0x434f2940 (LWP 28064)]
> [New Thread 0x42cf1940 (LWP 28063)]
> [New Thread 0x424f0940 (LWP 28062)]
> [New Thread 0x41cef940 (LWP 28061)]
> [New Thread 0x414ee940 (LWP 28060)]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/libnsl.so.1...done.
> Loaded symbols for /lib64/libnsl.so.1
> Reading symbols from /lib64/librt.so.1...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libresolv.so.2...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...done.
> Loaded symbols for /lib64/libnss_files.so.2
> 0x0000003095a0d524 in __lll_lock_wait () from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x0000003095a0d524 in __lll_lock_wait () from /lib64/libpthread.so.
> 0
> #1 0x0000003095a08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2 0x0000003095a08cdc in pthread_mutex_lock () from /lib64/
> libpthread.so.0
> #3 0x000000000040d08e in item_get (key=0x7f2cf8ac1424
> "ttt_newuser_2272938860", nkey=128) at thread.c:343
> #4 0x0000000000403c14 in process_get_command (c=0x7f2d3888f080,
> tokens=0x414eded0, ntokens=0, return_cas=false)
> at memcached.c:2542
> #5 0x0000000000408256 in process_command (c=0x7f2d3888f080,
> command=<value optimized out>) at memcached.c:2975
> #6 0x0000000000408bfe in try_read_command (c=0x7f2d3888f080) at
> memcached.c:3185
> #7 0x000000000040989d in event_handler (fd=<value optimized out>,
> which=128, arg=0x7f2d3888f080) at memcached.c:3491
> #8 0x00007f2ec05c25f8 in event_base_loop (base=0x63f420, flags=0) at
> event.c:392
> #9 0x000000000040cbb4 in worker_libevent (arg=0x633f70) at thread.c:
> 245
> #10 0x0000003095a0673d in start_thread () from /lib64/libpthread.so.0
> #11 0x00000030952d44bd in clone () from /lib64/libc.so.6
>
> gdb -p 28061
> GNU gdb Fedora (6.8-37.el5)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
> gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Attaching to process 28061
>
> warning: process 28061 is a cloned process
> Reading symbols from /usr/local/sinawap/bin/memcached...done.
> Reading symbols from /usr/local/sinawap/lib/libevent-1.4.so.2...done.
> Loaded symbols for /usr/local/sinawap/lib/libevent-1.4.so.2
> Reading symbols from /lib64/libpthread.so.0...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 0x7f2ec05b86e0 (LWP 28059)]
> [New Thread 0x434f2940 (LWP 28064)]
> [New Thread 0x42cf1940 (LWP 28063)]
> [New Thread 0x424f0940 (LWP 28062)]
> [New Thread 0x41cef940 (LWP 28061)]
> [New Thread 0x414ee940 (LWP 28060)]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/libnsl.so.1...done.
> Loaded symbols for /lib64/libnsl.so.1
> Reading symbols from /lib64/librt.so.1...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libresolv.so.2...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...done.
> Loaded symbols for /lib64/libnss_files.so.2
> 0x0000003095a0d524 in __lll_lock_wait () from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x0000003095a0d524 in __lll_lock_wait () from /lib64/libpthread.so.
> 0
> #1 0x0000003095a08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2 0x0000003095a08cdc in pthread_mutex_lock () from /lib64/
> libpthread.so.0
> #3 0x000000000040d0f6 in item_alloc (key=0x7f2cd3a2a0a4
> "ttt_api_unread_2557924625", nkey=128, flags=0,
> exptime=4294967295, nbytes=6395456) at thread.c:331
> #4 0x0000000000407fa7 in process_update_command (c=0x7f2ebc16e900,
> tokens=<value optimized out>,
> ntokens=<value optimized out>, comm=2, handle_cas=false) at
> memcached.c:2728
> #5 0x00000000004083be in process_command (c=0x7f2ebc16e900,
> command=<value optimized out>) at memcached.c:2984
> #6 0x0000000000408bfe in try_read_command (c=0x7f2ebc16e900) at
> memcached.c:3185
> #7 0x000000000040989d in event_handler (fd=<value optimized out>,
> which=128, arg=0x7f2ebc16e900) at memcached.c:3491
> #8 0x00007f2ec05c25f8 in event_base_loop (base=0x658130, flags=0) at
> event.c:392
> #9 0x000000000040cbb4 in worker_libevent (arg=0x636c98) at thread.c:
> 245
> #10 0x0000003095a0673d in start_thread () from /lib64/libpthread.so.0
> #11 0x00000030952d44bd in clone () from /lib64/libc.so.6
>
> gdb -p 28062
> GNU gdb Fedora (6.8-37.el5)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
> gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Attaching to process 28062
>
> warning: process 28062 is a cloned process
> Reading symbols from /usr/local/sinawap/bin/memcached...done.
> Reading symbols from /usr/local/sinawap/lib/libevent-1.4.so.2...done.
> Loaded symbols for /usr/local/sinawap/lib/libevent-1.4.so.2
> Reading symbols from /lib64/libpthread.so.0...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 0x7f2ec05b86e0 (LWP 28059)]
> [New Thread 0x434f2940 (LWP 28064)]
> [New Thread 0x42cf1940 (LWP 28063)]
> [New Thread 0x424f0940 (LWP 28062)]
> [New Thread 0x41cef940 (LWP 28061)]
> [New Thread 0x414ee940 (LWP 28060)]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/libnsl.so.1...done.
> Loaded symbols for /lib64/libnsl.so.1
> Reading symbols from /lib64/librt.so.1...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libresolv.so.2...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...done.
> Loaded symbols for /lib64/libnss_files.so.2
> 0x0000003095a0d524 in __lll_lock_wait () from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x0000003095a0d524 in __lll_lock_wait () from /lib64/libpthread.so.
> 0
> #1 0x0000003095a08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2 0x0000003095a08cdc in pthread_mutex_lock () from /lib64/
> libpthread.so.0
> #3 0x000000000040d08e in item_get (key=0x7f2cdedceb84
> "iphone_2283873065_2012042520", nkey=128) at thread.c:343
> #4 0x0000000000403c14 in process_get_command (c=0x7f2d3861ff20,
> tokens=0x424efed0, ntokens=0, return_cas=false)
> at memcached.c:2542
> #5 0x0000000000408256 in process_command (c=0x7f2d3861ff20,
> command=<value optimized out>) at memcached.c:2975
> #6 0x0000000000408bfe in try_read_command (c=0x7f2d3861ff20) at
> memcached.c:3185
> #7 0x000000000040989d in event_handler (fd=<value optimized out>,
> which=128, arg=0x7f2d3861ff20) at memcached.c:3491
> #8 0x00007f2ec05c25f8 in event_base_loop (base=0x690ed0, flags=0) at
> event.c:392
> #9 0x000000000040cbb4 in worker_libevent (arg=0x6399c0) at thread.c:
> 245
> #10 0x0000003095a0673d in start_thread () from /lib64/libpthread.so.0
> #11 0x00000030952d44bd in clone () from /lib64/libc.so.6
>
> gdb -p 28063
> GNU gdb Fedora (6.8-37.el5)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
> gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Attaching to process 28063
>
> warning: process 28063 is a cloned process
> Reading symbols from /usr/local/sinawap/bin/memcached...done.
> Reading symbols from /usr/local/sinawap/lib/libevent-1.4.so.2...done.
> Loaded symbols for /usr/local/sinawap/lib/libevent-1.4.so.2
> Reading symbols from /lib64/libpthread.so.0...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 0x7f2ec05b86e0 (LWP 28059)]
> [New Thread 0x434f2940 (LWP 28064)]
> [New Thread 0x42cf1940 (LWP 28063)]
> [New Thread 0x424f0940 (LWP 28062)]
> [New Thread 0x41cef940 (LWP 28061)]
> [New Thread 0x414ee940 (LWP 28060)]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/libnsl.so.1...done.
> Loaded symbols for /lib64/libnsl.so.1
> Reading symbols from /lib64/librt.so.1...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libresolv.so.2...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...done.
> Loaded symbols for /lib64/libnss_files.so.2
> 0x000000000040c2d1 in assoc_find (key=0x7f2d231264f4
> "TC_db784e309e27b22873f62c3329d032c9", nkey=35) at assoc.c:85
> 85 if ((nkey == it->nkey) && (memcmp(key, ITEM_key(it),
> nkey) == 0)) {
> (gdb) bt
> #0 0x000000000040c2d1 in assoc_find (key=0x7f2d231264f4
> "TC_db784e309e27b22873f62c3329d032c9", nkey=35) at assoc.c:85
> #1 0x000000000040ba1e in do_item_get (key=0xbef4bc56 <Address
> 0xbef4bc56 out of bounds>, nkey=3432596860) at items.c:466
> #2 0x000000000040d099 in item_get (key=0x7f2d231264f4
> "TC_db784e309e27b22873f62c3329d032c9", nkey=35) at thread.c:344
> #3 0x0000000000403c14 in process_get_command (c=0x7f2d6227cb00,
> tokens=0x42cf0ed0, ntokens=25009664, return_cas=false)
> at memcached.c:2542
> #4 0x0000000000408256 in process_command (c=0x7f2d6227cb00,
> command=<value optimized out>) at memcached.c:2975
> #5 0x0000000000408bfe in try_read_command (c=0x7f2d6227cb00) at
> memcached.c:3185
> #6 0x000000000040989d in event_handler (fd=<value optimized out>,
> which=17788, arg=0x7f2d6227cb00) at memcached.c:3491
> #7 0x00007f2ec05c25f8 in event_base_loop (base=0x6a9c70, flags=0) at
> event.c:392
> #8 0x000000000040cbb4 in worker_libevent (arg=0x63c6e8) at thread.c:
> 245
> #9 0x0000003095a0673d in start_thread () from /lib64/libpthread.so.0
> #10 0x00000030952d44bd in clone () from /lib64/libc.so.6
>
>
> gdb -p 28064
> GNU gdb Fedora (6.8-37.el5)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
> gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Attaching to process 28064
>
> warning: process 28064 is a cloned process
> Reading symbols from /usr/local/sinawap/bin/memcached...done.
> Reading symbols from /usr/local/sinawap/lib/libevent-1.4.so.2...done.
> Loaded symbols for /usr/local/sinawap/lib/libevent-1.4.so.2
> Reading symbols from /lib64/libpthread.so.0...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 0x7f2ec05b86e0 (LWP 28059)]
> [New Thread 0x434f2940 (LWP 28064)]
> [New Thread 0x42cf1940 (LWP 28063)]
> [New Thread 0x424f0940 (LWP 28062)]
> [New Thread 0x41cef940 (LWP 28061)]
> [New Thread 0x414ee940 (LWP 28060)]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/libnsl.so.1...done.
> Loaded symbols for /lib64/libnsl.so.1
> Reading symbols from /lib64/librt.so.1...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libresolv.so.2...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...done.
> Loaded symbols for /lib64/libnss_files.so.2
> 0x0000003095a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/
> libpthread.so.0
> (gdb) bt
> #0 0x0000003095a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /
> lib64/libpthread.so.0
> #1 0x000000000040c10c in assoc_maintenance_thread (arg=<value
> optimized out>) at assoc.c:223
> #2 0x0000003095a0673d in start_thread () from /lib64/libpthread.so.0
> #3 0x00000030952d44bd in clone () from /lib64/libc.so.6
>
