>> ldapsearch hangs. Dirsrv is not responding now.
> if the  server is hanging, can you get a pstack
>> 

Thread 45 (Thread 0x7fc6a562d700 (LWP 1868)):
#0  0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
#1  0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fc6a961f987 in deadlock_threadmain () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#4  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 44 (Thread 0x7fc6a4e2c700 (LWP 1869)):
#0  0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
#1  0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fc6a9623a4e in checkpoint_threadmain () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#4  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 43 (Thread 0x7fc6a462b700 (LWP 1870)):
#0  0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
#1  0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fc6a961fc0f in trickle_threadmain () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#4  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 42 (Thread 0x7fc6a3e2a700 (LWP 1871)):
#0  0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
#1  0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fc6a961a667 in perf_threadmain () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#4  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 41 (Thread 0x7fc6a3629700 (LWP 1900)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5482c68 in slapi_wait_condvar () from 
/usr/lib64/dirsrv/libslapd.so.0
#3  0x00007fc6abd455be in cos_cache_wait_on_change () from 
/usr/lib64/dirsrv/plugins/libcos-plugin.so
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 40 (Thread 0x7fc6b5735700 (LWP 1901)):
#0  0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
#2  0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
#3  0x00007fc6a937be84 in _cl5TrimMain () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 39 (Thread 0x7fc6a2e28700 (LWP 1902)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6a93928d4 in protocol_sleep () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#3  0x00007fc6a93949e5 in repl5_inc_run () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#4  0x00007fc6a9399421 in prot_thread_main () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#5  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#6  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#7  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 38 (Thread 0x7fc6a2627700 (LWP 1903)):
#0  0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
#2  0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
#3  0x00007fc6a93928d4 in protocol_sleep () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#4  0x00007fc6a9395158 in repl5_inc_run () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#5  0x00007fc6a9399421 in prot_thread_main () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#6  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#7  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#8  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 37 (Thread 0x7fc6a1a04700 (LWP 1906)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5482c68 in slapi_wait_condvar () from 
/usr/lib64/dirsrv/libslapd.so.0
#3  0x00007fc6a7cbef5d in roles_cache_wait_on_change () from 
/usr/lib64/dirsrv/plugins/libroles-plugin.so
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 36 (Thread 0x7fc6a1203700 (LWP 1907)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5482c68 in slapi_wait_condvar () from 
/usr/lib64/dirsrv/libslapd.so.0
#3  0x00007fc6a7cbef5d in roles_cache_wait_on_change () from 
/usr/lib64/dirsrv/plugins/libroles-plugin.so
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7fc6a0a02700 (LWP 1908)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5482c68 in slapi_wait_condvar () from 
/usr/lib64/dirsrv/libslapd.so.0
#3  0x00007fc6a7cbef5d in roles_cache_wait_on_change () from 
/usr/lib64/dirsrv/plugins/libroles-plugin.so
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7fc693fff700 (LWP 1909)):
#0  0x00007fc6b31ed939 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3841ef8 in pt_TimedWait () from /lib64/libnspr4.so
#2  0x00007fc6b38423be in PR_WaitCondVar () from /lib64/libnspr4.so
#3  0x00007fc6b593a773 in housecleaning ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7fc6937fe700 (LWP 1910)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
#2  0x00007fc6a9622456 in dblayer_txn_begin () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6a965d615 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#4  0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#5  0x00007fc6b544f387 in modify_internal_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#6  0x00007fc6a939f76f in replica_write_ruv () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#7  0x00007fc6a939f9fe in replica_update_state () from 
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#8  0x00007fc6b542965a in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0
#9  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#10 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#11 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fc692924700 (LWP 1912)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fc692123700 (LWP 1913)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fc691922700 (LWP 1914)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
#2  0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
#3  0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
#4  0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
#5  0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
#6  0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
#7  0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
#8  0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
#9  0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
#10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
#11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
#12 0x00007fc6a962e6de in idl_new_fetch () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#13 0x00007fc6a963cc2b in index_read_ext_allids () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#14 0x00007fc6a96272ed in keys2idl () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#15 0x00007fc6a9627afe in ava_candidates.isra () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#16 0x00007fc6a96280fa in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#17 0x00007fc6a96290ce in list_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#18 0x00007fc6a9628062 in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#19 0x00007fc6a96631a7 in subtree_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#20 0x00007fc6a9664811 in ldbm_back_search () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#21 0x00007fc6b5455410 in op_shared_search () from 
/usr/lib64/dirsrv/libslapd.so.0
#22 0x00007fc6b5943fc6 in do_search ()
#23 0x00007fc6b5932b4c in connection_threadmain ()
#24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fc691121700 (LWP 1915)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
#2  0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
#3  0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
#4  0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
#5  0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
#6  0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
#7  0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
#8  0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
#9  0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
#10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
#11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
#12 0x00007fc6a962e6de in idl_new_fetch () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#13 0x00007fc6a963cc2b in index_read_ext_allids () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#14 0x00007fc6a96272ed in keys2idl () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#15 0x00007fc6a9627afe in ava_candidates.isra () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#16 0x00007fc6a96280fa in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#17 0x00007fc6a96290ce in list_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#18 0x00007fc6a9628062 in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#19 0x00007fc6a96631a7 in subtree_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#20 0x00007fc6a9664811 in ldbm_back_search () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#21 0x00007fc6b5455410 in op_shared_search () from 
/usr/lib64/dirsrv/libslapd.so.0
#22 0x00007fc6b5943fc6 in do_search ()
#23 0x00007fc6b5932b4c in connection_threadmain ()
#24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fc690920700 (LWP 1916)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
#2  0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
#3  0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
#4  0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
#5  0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
#6  0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
#7  0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
#8  0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
#9  0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
#10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
#11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
#12 0x00007fc6a962e6de in idl_new_fetch () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#13 0x00007fc6a963cc2b in index_read_ext_allids () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#14 0x00007fc6a96272ed in keys2idl () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#15 0x00007fc6a9627afe in ava_candidates.isra () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#16 0x00007fc6a96280fa in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#17 0x00007fc6a96290ce in list_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#18 0x00007fc6a9628062 in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#19 0x00007fc6a96631a7 in subtree_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#20 0x00007fc6a9664811 in ldbm_back_search () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#21 0x00007fc6b5455410 in op_shared_search () from 
/usr/lib64/dirsrv/libslapd.so.0
#22 0x00007fc6b5943fc6 in do_search ()
#23 0x00007fc6b5932b4c in connection_threadmain ()
#24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fc67ffff700 (LWP 1917)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fc67f7fe700 (LWP 1918)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
#2  0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
#3  0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
#4  0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
#5  0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
#6  0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so
#7  0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
#8  0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
#9  0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
#10 0x00007fc6add50ced in __db_get () from /lib64/libdb-5.3.so
#11 0x00007fc6add5473b in __db_get_pp () from /lib64/libdb-5.3.so
#12 0x00007fc6a962a9bb in id2entry () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#13 0x00007fc6a9626cbd in dn2entry_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#14 0x00007fc6a9629bb1 in find_entry_internal.isra () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#15 0x00007fc6a9629e99 in find_entry () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#16 0x00007fc6a9663b5b in ldbm_back_search () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#17 0x00007fc6b5455410 in op_shared_search () from 
/usr/lib64/dirsrv/libslapd.so.0
#18 0x00007fc6b546553e in search_internal_callback_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#19 0x00007fc6b54657c8 in search_internal_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#20 0x00007fc6b5465b73 in slapi_search_internal_get_entry () from 
/usr/lib64/dirsrv/libslapd.so.0
#21 0x00007fc6aad02234 in ipalockout_preop () from 
/usr/lib64/dirsrv/plugins/libipa_lockout.so
#22 0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#23 0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#24 0x00007fc6b592c3d8 in do_bind ()
#25 0x00007fc6b5932974 in connection_threadmain ()
#26 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#27 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#28 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fc67effd700 (LWP 1919)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fc67e7fc700 (LWP 1920)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fc67dffb700 (LWP 1921)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fc67d7fa700 (LWP 1922)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
#2  0x00007fc6a9622456 in dblayer_txn_begin () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6a965d615 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#4  0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#5  0x00007fc6b544f387 in modify_internal_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#6  0x00007fc6aad02bd3 in ipalockout_postop () from 
/usr/lib64/dirsrv/plugins/libipa_lockout.so
#7  0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#8  0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#9  0x00007fc6b592c22d in do_bind ()
#10 0x00007fc6b5932974 in connection_threadmain ()
#11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fc67cff9700 (LWP 1923)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
#2  0x00007fc6a9622456 in dblayer_txn_begin () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6a965d615 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#4  0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#5  0x00007fc6b544f387 in modify_internal_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#6  0x00007fc6aad02bd3 in ipalockout_postop () from 
/usr/lib64/dirsrv/plugins/libipa_lockout.so
#7  0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#8  0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#9  0x00007fc6b592c22d in do_bind ()
#10 0x00007fc6b5932974 in connection_threadmain ()
#11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fc67c7f8700 (LWP 1924)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fc67bff7700 (LWP 1925)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
#2  0x00007fc6a9622456 in dblayer_txn_begin () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6a965d615 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#4  0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#5  0x00007fc6b544f387 in modify_internal_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#6  0x00007fc6aad02bd3 in ipalockout_postop () from 
/usr/lib64/dirsrv/plugins/libipa_lockout.so
#7  0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#8  0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#9  0x00007fc6b592c22d in do_bind ()
#10 0x00007fc6b5932974 in connection_threadmain ()
#11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fc67b7f6700 (LWP 1926)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fc67aff5700 (LWP 1927)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fc67a7f4700 (LWP 1928)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fc679ff3700 (LWP 1929)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fc6797f2700 (LWP 1930)):
#0  0x00007fc6b31eff1d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc6b31ea9be in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007fc6b38420b9 in PR_Lock () from /lib64/libnspr4.so
#3  0x00007fc6a7ec99b0 in retrocl_postob () from 
/usr/lib64/dirsrv/plugins/libretrocl-plugin.so
#4  0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#5  0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#6  0x00007fc6a965d231 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#7  0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#8  0x00007fc6b544f387 in modify_internal_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#9  0x00007fc6a8d354a2 in memberof_fix_memberof_callback () from 
/usr/lib64/dirsrv/plugins/libmemberof-plugin.so
#10 0x00007fc6a8d35f65 in memberof_modop_one_replace_r () from 
/usr/lib64/dirsrv/plugins/libmemberof-plugin.so
#11 0x00007fc6a8d362a6 in memberof_mod_smod_list () from 
/usr/lib64/dirsrv/plugins/libmemberof-plugin.so
#12 0x00007fc6a8d3758d in memberof_postop_modify () from 
/usr/lib64/dirsrv/plugins/libmemberof-plugin.so
#13 0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#14 0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#15 0x00007fc6a965d231 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#16 0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#17 0x00007fc6b544fbe7 in do_modify () from /usr/lib64/dirsrv/libslapd.so.0
#18 0x00007fc6b5932925 in connection_threadmain ()
#19 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#20 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#21 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fc678ff1700 (LWP 1931)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
#2  0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
#3  0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
#4  0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
#5  0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
#6  0x00007fc6adc9e4f0 in __bam_search () from /lib64/libdb-5.3.so
#7  0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
#8  0x00007fc6adc8b074 in __bamc_get () from /lib64/libdb-5.3.so
#9  0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
#10 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
#11 0x00007fc6a9672b70 in ldbm_back_seq () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#12 0x00007fc6b54650f9 in seq_internal_callback_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#13 0x00007fc6b54652a9 in slapi_seq_callback () from 
/usr/lib64/dirsrv/libslapd.so.0
#14 0x00007fc6a7ec8919 in retrocl_update_lastchangenumber () from 
/usr/lib64/dirsrv/plugins/libretrocl-plugin.so
#15 0x00007fc6a7ec89bb in retrocl_assign_changenumber () from 
/usr/lib64/dirsrv/plugins/libretrocl-plugin.so
#16 0x00007fc6a7ec99b5 in retrocl_postob () from 
/usr/lib64/dirsrv/plugins/libretrocl-plugin.so
#17 0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#18 0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#19 0x00007fc6a965d231 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#20 0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#21 0x00007fc6b544fbe7 in do_modify () from /usr/lib64/dirsrv/libslapd.so.0
#22 0x00007fc6b5932925 in connection_threadmain ()
#23 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#24 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#25 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fc6787f0700 (LWP 1932)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fc677fef700 (LWP 1933)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fc6777ee700 (LWP 1934)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
#2  0x00007fc6a9622456 in dblayer_txn_begin () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6a965d615 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#4  0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#5  0x00007fc6b544f387 in modify_internal_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#6  0x00007fc6aad02bd3 in ipalockout_postop () from 
/usr/lib64/dirsrv/plugins/libipa_lockout.so
#7  0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#8  0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#9  0x00007fc6b592c22d in do_bind ()
#10 0x00007fc6b5932974 in connection_threadmain ()
#11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fc676fed700 (LWP 1935)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b38426b3 in PR_EnterMonitor () from /lib64/libnspr4.so
#2  0x00007fc6a9622456 in dblayer_txn_begin () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007fc6a965d615 in ldbm_back_modify () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#4  0x00007fc6b544e905 in op_shared_modify () from 
/usr/lib64/dirsrv/libslapd.so.0
#5  0x00007fc6b544f387 in modify_internal_pb () from 
/usr/lib64/dirsrv/libslapd.so.0
#6  0x00007fc6aad02bd3 in ipalockout_postop () from 
/usr/lib64/dirsrv/plugins/libipa_lockout.so
#7  0x00007fc6b546156f in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#8  0x00007fc6b5461893 in plugin_call_plugins () from 
/usr/lib64/dirsrv/libslapd.so.0
#9  0x00007fc6b592c22d in do_bind ()
#10 0x00007fc6b5932974 in connection_threadmain ()
#11 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#12 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#13 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fc6767ec700 (LWP 1936)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fc675feb700 (LWP 1937)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
#2  0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
#3  0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
#4  0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
#5  0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
#6  0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
#7  0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
#8  0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
#9  0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
#10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
#11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
#12 0x00007fc6a962e6de in idl_new_fetch () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#13 0x00007fc6a963cc2b in index_read_ext_allids () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#14 0x00007fc6a96272ed in keys2idl () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#15 0x00007fc6a9627afe in ava_candidates.isra () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#16 0x00007fc6a96280fa in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#17 0x00007fc6a96290ce in list_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#18 0x00007fc6a9628062 in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#19 0x00007fc6a96631a7 in subtree_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#20 0x00007fc6a9664811 in ldbm_back_search () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#21 0x00007fc6b5455410 in op_shared_search () from 
/usr/lib64/dirsrv/libslapd.so.0
#22 0x00007fc6b5943fc6 in do_search ()
#23 0x00007fc6b5932b4c in connection_threadmain ()
#24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fc6757ea700 (LWP 1938)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fc674fe9700 (LWP 1939)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fc6747e8700 (LWP 1940)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6adc811fb in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
#2  0x00007fc6adc8059b in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
#3  0x00007fc6add2aa91 in __lock_get_internal () from /lib64/libdb-5.3.so
#4  0x00007fc6add2b657 in __lock_get () from /lib64/libdb-5.3.so
#5  0x00007fc6add576af in __db_lget () from /lib64/libdb-5.3.so
#6  0x00007fc6adc9d5d9 in __bam_get_root () from /lib64/libdb-5.3.so
#7  0x00007fc6adc9d91b in __bam_search () from /lib64/libdb-5.3.so
#8  0x00007fc6adc89144 in __bamc_search () from /lib64/libdb-5.3.so
#9  0x00007fc6adc8adff in __bamc_get () from /lib64/libdb-5.3.so
#10 0x00007fc6add43ba3 in __dbc_iget () from /lib64/libdb-5.3.so
#11 0x00007fc6add53092 in __dbc_get_pp () from /lib64/libdb-5.3.so
#12 0x00007fc6a962e6de in idl_new_fetch () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#13 0x00007fc6a963cc2b in index_read_ext_allids () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#14 0x00007fc6a96272ed in keys2idl () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#15 0x00007fc6a9627afe in ava_candidates.isra () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#16 0x00007fc6a96280fa in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#17 0x00007fc6a96290ce in list_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#18 0x00007fc6a9628062 in filter_candidates_ext () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#19 0x00007fc6a96631a7 in subtree_candidates () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#20 0x00007fc6a9664811 in ldbm_back_search () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#21 0x00007fc6b5455410 in op_shared_search () from 
/usr/lib64/dirsrv/libslapd.so.0
#22 0x00007fc6b5943fc6 in do_search ()
#23 0x00007fc6b5932b4c in connection_threadmain ()
#24 0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#25 0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#26 0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fc673fe7700 (LWP 1941)):
#0  0x00007fc6b31ed590 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fc6b3842440 in PR_WaitCondVar () from /lib64/libnspr4.so
#2  0x00007fc6b5930b9e in connection_wait_for_new_work ()
#3  0x00007fc6b5931dc9 in connection_threadmain ()
#4  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#5  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fc6737e6700 (LWP 1942)):
#0  0x00007fc6b2f1aae3 in select () from /lib64/libc.so.6
#1  0x00007fc6b5492a99 in DS_Sleep () from /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fc6b5933855 in time_thread ()
#3  0x00007fc6b3847cab in _pt_root () from /lib64/libnspr4.so
#4  0x00007fc6b31e852a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc6b2f2422d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fc6b58fb800 (LWP 1863)):
#0  0x00007fc6b2f18c8d in poll () from /lib64/libc.so.6
#1  0x00007fc6b3843da8 in _pr_poll_with_poll () from /lib64/libnspr4.so
#2  0x00007fc6b5936b11 in slapd_daemon ()
#3  0x00007fc6b59294f4 in main ()


>> This replica is on virtual machine (ganeti). We had problems with 
>> replication to vm, but after force-sync all was fine. On physical servers 
>> all works fine.
> the messages indicate there could be  many concurrent operations, because 
> individual ops are not fast enough, could your VM have less/slower resources 
> than the physical machines ?

VMs have less rsources and slower than physical machines.
VM: 4 cpu, hd via drbd on sas drives,

physical: more cores, faster (ssd) drives

Best regards,
Lukasz Jaworski 'Ender'



>> Lukasz Jaworski 'Ender'
>> 
>> Wiadomość napisana przez Ludwig Krispenz <lkris...@redhat.com> w dniu 6 maj 
>> 2015, o godz. 10:52:
>> 
>>> Hi,
>>> 
>>> there seem to be different issues,
>>> - I don't know what the ipactl status is looking for when it generates the 
>>> error message about no matching master,
>>> but I don't think it is related to the retro changelog.
>>> 
>>> - the retro changelog errors for adding and deleting
>>> -- the add failures are about aborted transactions because a page cannot be 
>>> accessed, this maybe caused by concurrent mods on different backends, which 
>>> want to update teh shared retro cl database.
>>> the changenumber reprted seems to be increasing, one error is about 
>>> changenumber 44975, the next about 45577, so it looks like changes into the 
>>> changelog are written and teh changenumber increases
>>> -- i'm not sure about the delete errors, but normally trimming would go on 
>>> after such an error message, the changenumber attempted to delete are 
>>> increasing.
>>> Could you verify which changes are in the changelog, and if these are 
>>> changing:
>>> ldapsearch -b "cn=changelog" dn
>>> 
>>> On 05/06/2015 09:52 AM, Łukasz Jaworski wrote:
>>>> Hi,
>>>> 
>>>> One of our replica hanged up morning. Error log after dirsrv restart:
>>>> [06/May/2015:09:28:15 +0200] - Retry count exceeded in delete
>>>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 38376 (rc: 51)
>>>> [06/May/2015:09:28:15 +0200] - Operation error fetching Null DN 
>>>> (6368aeb7-f3c111e4-ae70ce39-9b469c1f), error -30993.
>>>> [06/May/2015:09:28:15 +0200] - dn2entry_ext: Failed to get id for 
>>>> changenumber=44975,cn=changelog from entryrdn index (-30993)
>>>> [06/May/2015:09:28:15 +0200] - Operation error fetching 
>>>> changenumber=44975,cn=changelog (null), error -30993.
>>>> [06/May/2015:09:28:15 +0200] DSRetroclPlugin - replog: an error occured 
>>>> while adding change number 44975, dn = changenumber=44975,cn=changelog: 
>>>> Operations error.
>>>> [06/May/2015:09:28:15 +0200] retrocl-plugin - retrocl_postob: operation 
>>>> failure [1]
>>>> [06/May/2015:09:28:15 +0200] - ldbm_back_seq deadlock retry BAD 1601, 
>>>> err=0 BDB0062 Successful return: 0
>>>> [06/May/2015:09:30:03 +0200] - ldbm_back_seq deadlock retry BAD 1601, 
>>>> err=0 BDB0062 Successful return: 0
>>>> [06/May/2015:09:30:06 +0200] - Retry count exceeded in delete
>>>> [06/May/2015:09:30:06 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 39297 (rc: 51)
>>>> 
>>>> I did "re-initialize" from other replica.
>>>> 
>>>> Now ipactl doesn't work. Shows: Configured hostname 'replica09.local' does 
>>>> not match any master server in LDAP. On lists replica09 is exists (twice)
>>>> 
>>>> # ipactl status
>>>> Failed to get list of services to probe status!
>>>> Configured hostname 'replica09.local' does not match any master server in 
>>>> LDAP:
>>>> replica01.local
>>>> replica02.local
>>>> replica03.local
>>>> replica04.local
>>>> replica05.local
>>>> replica06.local
>>>> replica07.local
>>>> replica08.local
>>>> replica09.local
>>>> replica10.local
>>>> replica09.local
>>>> 
>>>> After dirsrv stop/start:
>>>> 
>>>> In error logs there are many:
>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 39290 (rc: 32)
>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 39291 (rc: 32)
>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 39292 (rc: 32)
>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 39293 (rc: 32)
>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 39294 (rc: 32)
>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 39295 (rc: 32)
>>>> [06/May/2015:09:50:30 +0200] DSRetroclPlugin - delete_changerecord: could 
>>>> not delete change record 39296 (rc: 32)
>>>> etc.
>>>> 
>>>> [06/May/2015:09:51:08 +0200] - Operation error fetching Null DN 
>>>> (9f51430a-f3c411e4-927ece39-9b469c1f), error -30993.
>>>> [06/May/2015:09:51:08 +0200] - dn2entry_ext: Failed to get id for 
>>>> changenumber=45577,cn=changelog from entryrdn index (-30993)
>>>> [06/May/2015:09:51:08 +0200] - Operation error fetching 
>>>> changenumber=45577,cn=changelog (null), error -30993.
>>>> [06/May/2015:09:51:08 +0200] DSRetroclPlugin - replog: an error occured 
>>>> while adding change number 45577, dn = changenumber=45577,cn=changelog: 
>>>> Operations error.
>>>> [06/May/2015:09:51:08 +0200] retrocl-plugin - retrocl_postob: operation 
>>>> failure [1]
>>>> [06/May/2015:09:51:08 +0200] - ldbm_back_seq deadlock retry BAD 1601, 
>>>> err=0 BDB0062 Successful return: 0
>>>> 
>>>> Packages:
>>>> freeipa-server-4.1.3-2.fc21.x86_64
>>>> 389-ds-base-1.3.3.8-1.fc21.x86_64
>>>> 389-ds-base-libs-1.3.3.8-1.fc21.x86_64
>>>> 
>>>> Best regards,
>>>> Ender
>>>> 
>>> -- 
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
> 


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to