>> ldapsearch hangs. Dirsrv is not responding now. > if the server is hanging, can you get a pstack >>
Thread 45 (Thread 0x7fc6a562d700 (LWP 1868)): #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6 #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007fc6a961f987 in deadlock_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7fc6a4e2c700 (LWP 1869)): #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6 #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007fc6a9623a4e in checkpoint_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7fc6a462b700 (LWP 1870)): #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6 #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007fc6a961fc0f in trickle_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7fc6a3e2a700 (LWP 1871)): #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6 #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007fc6a961a667 in perf_threadmain () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7fc6a3629700 (LWP 1900)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5482c68 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007fc6abd455be in cos_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libcos-plugin.so #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7fc6b5735700 (LWP 1901)): #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007fc6a937be84 in _cl5TrimMain () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7fc6a2e28700 (LWP 1902)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6a93928d4 in protocol_sleep () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #3 0x00007fc6a93949e5 in repl5_inc_run () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #4 0x00007fc6a9399421 in prot_thread_main () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #5 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #6 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7fc6a2627700 (LWP 1903)): #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007fc6a93928d4 in protocol_sleep () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #4 0x00007fc6a9395158 in repl5_inc_run () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #5 0x00007fc6a9399421 in prot_thread_main () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #6 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #7 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7fc6a1a04700 (LWP 1906)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5482c68 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7fc6a1203700 (LWP 1907)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5482c68 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fc6a0a02700 (LWP 1908)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5482c68 in slapi_wait_condvar () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00007fc6a7cbef5d in roles_cache_wait_on_change () from /usr/lib64/dirsrv/plugins/libroles-plugin.so #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fc693fff700 (LWP 1909)): #0 0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so #2 0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so #3 0x00007fc6b593a773 in housecleaning () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fc6937fe700 (LWP 1910)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007fc6a939f76f in replica_write_ruv () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #7 0x00007fc6a939f9fe in replica_update_state () from /usr/lib64/dirsrv/plugins/libreplication-plugin.so #8 0x00007fc6b542965a in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #10 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #11 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fc692924700 (LWP 1912)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fc692123700 (LWP 1913)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fc691922700 (LWP 1914)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007fc6b5943fc6 in do_search () #23 0x00007fc6b5932b4c in connection_threadmain () #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fc691121700 (LWP 1915)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007fc6b5943fc6 in do_search () #23 0x00007fc6b5932b4c in connection_threadmain () #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fc690920700 (LWP 1916)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007fc6b5943fc6 in do_search () #23 0x00007fc6b5932b4c in connection_threadmain () #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fc67ffff700 (LWP 1917)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fc67f7fe700 (LWP 1918)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so #6 0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007fc6add50ced in __db_get () from /lib64/libdb-5.3.so #11 0x00007fc6add5473b in __db_get_pp () from /lib64/libdb-5.3.so #12 0x00007fc6a962a9bb in id2entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007fc6a9626cbd in dn2entry_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007fc6a9629bb1 in find_entry_internal.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007fc6a9629e99 in find_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007fc6a9663b5b in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #18 0x00007fc6b546553e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 #19 0x00007fc6b54657c8 in search_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #20 0x00007fc6b5465b73 in slapi_search_internal_get_entry () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007fc6aad02234 in ipalockout_preop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #22 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #23 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #24 0x00007fc6b592c3d8 in do_bind () #25 0x00007fc6b5932974 in connection_threadmain () #26 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #27 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #28 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fc67effd700 (LWP 1919)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fc67e7fc700 (LWP 1920)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fc67dffb700 (LWP 1921)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fc67d7fa700 (LWP 1922)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007fc6b592c22d in do_bind () #10 0x00007fc6b5932974 in connection_threadmain () #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fc67cff9700 (LWP 1923)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007fc6b592c22d in do_bind () #10 0x00007fc6b5932974 in connection_threadmain () #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fc67c7f8700 (LWP 1924)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fc67bff7700 (LWP 1925)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007fc6b592c22d in do_bind () #10 0x00007fc6b5932974 in connection_threadmain () #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fc67b7f6700 (LWP 1926)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fc67aff5700 (LWP 1927)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fc67a7f4700 (LWP 1928)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fc679ff3700 (LWP 1929)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fc6797f2700 (LWP 1930)): #0 0x00007fc6b31eff1d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc6b31ea9be in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00007fc6b38420b9 in PR_Lock () from /lib64/libnspr4.so #3 0x00007fc6a7ec99b0 in retrocl_postob () from /usr/lib64/dirsrv/plugins/libretrocl-plugin.so #4 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007fc6a965d231 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #7 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007fc6a8d354a2 in memberof_fix_memberof_callback () from /usr/lib64/dirsrv/plugins/libmemberof-plugin.so #10 0x00007fc6a8d35f65 in memberof_modop_one_replace_r () from /usr/lib64/dirsrv/plugins/libmemberof-plugin.so #11 0x00007fc6a8d362a6 in memberof_mod_smod_list () from /usr/lib64/dirsrv/plugins/libmemberof-plugin.so #12 0x00007fc6a8d3758d in memberof_postop_modify () from /usr/lib64/dirsrv/plugins/libmemberof-plugin.so #13 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #14 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #15 0x00007fc6a965d231 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #17 0x00007fc6b544fbe7 in do_modify () from /usr/lib64/dirsrv/libslapd.so.0 #18 0x00007fc6b5932925 in connection_threadmain () #19 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #20 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc678ff1700 (LWP 1931)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so #6 0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so #7 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so #8 0x00007fc6adc8b074 in __bamc_get () from /lib64/libdb-5.3.so #9 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so #10 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so #11 0x00007fc6a9672b70 in ldbm_back_seq () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #12 0x00007fc6b54650f9 in seq_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0 #13 0x00007fc6b54652a9 in slapi_seq_callback () from /usr/lib64/dirsrv/libslapd.so.0 #14 0x00007fc6a7ec8919 in retrocl_update_lastchangenumber () from /usr/lib64/dirsrv/plugins/libretrocl-plugin.so #15 0x00007fc6a7ec89bb in retrocl_assign_changenumber () from /usr/lib64/dirsrv/plugins/libretrocl-plugin.so #16 0x00007fc6a7ec99b5 in retrocl_postob () from /usr/lib64/dirsrv/plugins/libretrocl-plugin.so #17 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #18 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #19 0x00007fc6a965d231 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #21 0x00007fc6b544fbe7 in do_modify () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007fc6b5932925 in connection_threadmain () #23 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #24 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #25 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc6787f0700 (LWP 1932)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc677fef700 (LWP 1933)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc6777ee700 (LWP 1934)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007fc6b592c22d in do_bind () #10 0x00007fc6b5932974 in connection_threadmain () #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc676fed700 (LWP 1935)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007fc6a9622456 in dblayer_txn_begin () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #3 0x00007fc6a965d615 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #4 0x00007fc6b544e905 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00007fc6b544f387 in modify_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00007fc6aad02bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #7 0x00007fc6b546156f in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00007fc6b5461893 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0 #9 0x00007fc6b592c22d in do_bind () #10 0x00007fc6b5932974 in connection_threadmain () #11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc6767ec700 (LWP 1936)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc675feb700 (LWP 1937)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007fc6b5943fc6 in do_search () #23 0x00007fc6b5932b4c in connection_threadmain () #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc6757ea700 (LWP 1938)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc674fe9700 (LWP 1939)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc6747e8700 (LWP 1940)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so #2 0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so #3 0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so #4 0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so #5 0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so #6 0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so #7 0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so #8 0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so #9 0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so #10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so #11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so #12 0x00007fc6a962e6de in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #13 0x00007fc6a963cc2b in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #14 0x00007fc6a96272ed in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #15 0x00007fc6a9627afe in ava_candidates.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #16 0x00007fc6a96280fa in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #17 0x00007fc6a96290ce in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #18 0x00007fc6a9628062 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #19 0x00007fc6a96631a7 in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #20 0x00007fc6a9664811 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #21 0x00007fc6b5455410 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #22 0x00007fc6b5943fc6 in do_search () #23 0x00007fc6b5932b4c in connection_threadmain () #24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc673fe7700 (LWP 1941)): #0 0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so #2 0x00007fc6b5930b9e in connection_wait_for_new_work () #3 0x00007fc6b5931dc9 in connection_threadmain () #4 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #5 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc6737e6700 (LWP 1942)): #0 0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6 #1 0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00007fc6b5933855 in time_thread () #3 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so #4 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc6b58fb800 (LWP 1863)): #0 0x00007fc6b2f18c8d in poll () from /lib64/libc.so.6 #1 0x00007fc6b3843da8 in _pr_poll_with_poll () from /lib64/libnspr4.so #2 0x00007fc6b5936b11 in slapd_daemon () #3 0x00007fc6b59294f4 in main () >> This replica is on virtual machine (ganeti). We had problems with >> replication to vm, but after force-sync all was fine. On physical servers >> all works fine. > the messages indicate there could be many concurrent operations, because > individual ops are not fast enough, could your VM have less/slower resources > than the physical machines ? VMs have less rsources and slower than physical machines. VM: 4 cpu, hd via drbd on sas drives, physical: more cores, faster (ssd) drives Best regards, Lukasz Jaworski 'Ender' >> Lukasz Jaworski 'Ender' >> >> Wiadomość napisana przez Ludwig Krispenz <lkris...@redhat.com> w dniu 6 maj >> 2015, o godz. 10:52: >> >>> Hi, >>> >>> there seem to be different issues, >>> - I don't know what the ipactl status is looking for when it generates the >>> error message about no matching master, >>> but I don't think it is related to the retro changelog. >>> >>> - the retro changelog errors for adding and deleting >>> -- the add failures are about aborted transactions because a page cannot be >>> accessed, this maybe caused by concurrent mods on different backends, which >>> want to update teh shared retro cl database. >>> the changenumber reprted seems to be increasing, one error is about >>> changenumber 44975, the next about 45577, so it looks like changes into the >>> changelog are written and teh changenumber increases >>> -- i'm not sure about the delete errors, but normally trimming would go on >>> after such an error message, the changenumber attempted to delete are >>> increasing. >>> Could you verify which changes are in the changelog, and if these are >>> changing: >>> ldapsearch -b "cn=changelog" dn >>> >>> On 05/06/2015 09:52 AM, Łukasz Jaworski wrote: >>>> Hi, >>>> >>>> One of our replica hanged up morning. Error log after dirsrv restart: >>>> [06/May/2015:09:28:15 +0200] - Retry count exceeded in delete >>>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 38376 (rc: 51) >>>> [06/May/2015:09:28:15 +0200] - Operation error fetching Null DN >>>> (6368aeb7-f3c111e4-ae70ce39-9b469c1f), error -30993. >>>> [06/May/2015:09:28:15 +0200] - dn2entry_ext: Failed to get id for >>>> changenumber=44975,cn=changelog from entryrdn index (-30993) >>>> [06/May/2015:09:28:15 +0200] - Operation error fetching >>>> changenumber=44975,cn=changelog (null), error -30993. >>>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - replog: an error occured >>>> while adding change number 44975, dn = changenumber=44975,cn=changelog: >>>> Operations error. >>>> [06/May/2015:09:28:15 +0200] retrocl-plugin - retrocl_postob: operation >>>> failure [1] >>>> [06/May/2015:09:28:15 +0200] - ldbm_back_seq deadlock retry BAD 1601, >>>> err=0 BDB0062 Successful return: 0 >>>> [06/May/2015:09:30:03 +0200] - ldbm_back_seq deadlock retry BAD 1601, >>>> err=0 BDB0062 Successful return: 0 >>>> [06/May/2015:09:30:06 +0200] - Retry count exceeded in delete >>>> [06/May/2015:09:30:06 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 39297 (rc: 51) >>>> >>>> I did "re-initialize" from other replica. >>>> >>>> Now ipactl doesn't work. Shows: Configured hostname 'replica09.local' does >>>> not match any master server in LDAP. On lists replica09 is exists (twice) >>>> >>>> # ipactl status >>>> Failed to get list of services to probe status! >>>> Configured hostname 'replica09.local' does not match any master server in >>>> LDAP: >>>> replica01.local >>>> replica02.local >>>> replica03.local >>>> replica04.local >>>> replica05.local >>>> replica06.local >>>> replica07.local >>>> replica08.local >>>> replica09.local >>>> replica10.local >>>> replica09.local >>>> >>>> After dirsrv stop/start: >>>> >>>> In error logs there are many: >>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 39290 (rc: 32) >>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 39291 (rc: 32) >>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 39292 (rc: 32) >>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 39293 (rc: 32) >>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 39294 (rc: 32) >>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 39295 (rc: 32) >>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could >>>> not delete change record 39296 (rc: 32) >>>> etc. >>>> >>>> [06/May/2015:09:51:08 +0200] - Operation error fetching Null DN >>>> (9f51430a-f3c411e4-927ece39-9b469c1f), error -30993. >>>> [06/May/2015:09:51:08 +0200] - dn2entry_ext: Failed to get id for >>>> changenumber=45577,cn=changelog from entryrdn index (-30993) >>>> [06/May/2015:09:51:08 +0200] - Operation error fetching >>>> changenumber=45577,cn=changelog (null), error -30993. >>>> [06/May/2015:09:51:08 +0200] DSRetroclPlugin - replog: an error occured >>>> while adding change number 45577, dn = changenumber=45577,cn=changelog: >>>> Operations error. >>>> [06/May/2015:09:51:08 +0200] retrocl-plugin - retrocl_postob: operation >>>> failure [1] >>>> [06/May/2015:09:51:08 +0200] - ldbm_back_seq deadlock retry BAD 1601, >>>> err=0 BDB0062 Successful return: 0 >>>> >>>> Packages: >>>> freeipa-server-4.1.3-2.fc21.x86_64 >>>> 389-ds-base-1.3.3.8-1.fc21.x86_64 >>>> 389-ds-base-libs-1.3.3.8-1.fc21.x86_64 >>>> >>>> Best regards, >>>> Ender >>>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
