This runs about 1/4 the load of the ones that had an issue. I don't know if I can run this up to see it. ________________________________ From: William Brown <[email protected]> Sent: Sunday, May 31, 2020 9:27:58 PM To: [email protected] <[email protected]> Subject: [EXTERNAL] [389-users] Re: Re: Re: new server setup hanging
> On 1 Jun 2020, at 11:43, Crocker, Deborah <[email protected]> wrote: > > Is this sufficient? Again, this server has a light load and we don't think > we saw the problem, although I do note that the CPU usage seems pretty high > for such a light load. All threads are idle except thread 1 that is checking if there are new connections. I don't see anything obviously wrong here ... :( > > > Thread 26 (Thread 0x7f0600d32700 (LWP 11330)): > #0 0x00007f06420a09a3 in select () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f06452e0649 in DS_Sleep () > at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f063a136bf7 in deadlock_threadmain () > at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5b00) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #4 0x00007f06429fdea5 in start_thread (arg=0x7f0600d32700) > at pthread_create.c:307 > #5 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 25 (Thread 0x7f0600531700 (LWP 11331)): > #0 0x00007f06420a09a3 in select () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f06452e0649 in DS_Sleep () > at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f063a13a7c7 in checkpoint_threadmain () > at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f59e0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #4 0x00007f06429fdea5 in start_thread (arg=0x7f0600531700) > at pthread_create.c:307 > #5 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 24 (Thread 0x7f05ffd30700 (LWP 11332)): > #0 0x00007f06420a09a3 in select () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f06452e0649 in DS_Sleep () > at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f063a136e47 in trickle_threadmain () > at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5c20) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #4 0x00007f06429fdea5 in start_thread (arg=0x7f05ffd30700) > at pthread_create.c:307 > #5 0x00007f06420a98dd in clone () > ---Type <return> to continue, or q <return> to quit--- > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 23 (Thread 0x7f05ff52f700 (LWP 11333)): > #0 0x00007f06420a09a3 in select () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f06452e0649 in DS_Sleep () > at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f063a1319f7 in perf_threadmain () > at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5440) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #4 0x00007f06429fdea5 in start_thread (arg=0x7f05ff52f700) > at pthread_create.c:307 > #5 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 22 (Thread 0x7f05fed2e700 (LWP 11334)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55e900, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x00007f06452ccf58 in slapi_wait_condvar () > at /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f063abe515e in cos_cache_wait_on_change () > at /usr/lib64/dirsrv/plugins/libcos-plugin.so > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b680) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fed2e700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 21 (Thread 0x7f05fe52d700 (LWP 11335)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55ebc0, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x00007f06452ccf58 in slapi_wait_condvar () > at /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f06382041fd in roles_cache_wait_on_change () > at /usr/lib64/dirsrv/plugins/libroles-plugin.so > ---Type <return> to continue, or q <return> to quit--- > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b440) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fe52d700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 20 (Thread 0x7f05fdd2c700 (LWP 11336)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b404d3c40, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x00007f06452ccf58 in slapi_wait_condvar () > at /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f06382041fd in roles_cache_wait_on_change () > at /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b320) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fdd2c700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 19 (Thread 0x7f05fd52b700 (LWP 11337)): > #0 0x00007f0642a01de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 > #1 0x00007f0643057d07 in pt_TimedWait (cv=cv@entry=0x557b3b55ec08, > ml=0x557b4055c160, timeout=timeout@entry=30000) > at ../../../nspr/pr/src/pthreads/ptsynch.c:258 > #2 0x00007f06430581ee in PR_WaitCondVar (cvar=0x557b3b55ec00, timeout=30000) > at ../../../nspr/pr/src/pthreads/ptsynch.c:387 > #3 0x0000557b3a0be208 in housecleaning () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b0e0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fd52b700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 18 (Thread 0x7f05fcd2a700 (LWP 11338)): > ---Type <return> to continue, or q <return> to quit--- > #0 0x00007f0642a01de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 > #1 0x00007f0643057d07 in pt_TimedWait (cv=cv@entry=0x557b3b61fa08, > ml=0x557b3b6709a0, timeout=timeout@entry=10000) > at ../../../nspr/pr/src/pthreads/ptsynch.c:258 > #2 0x00007f06430581ee in PR_WaitCondVar (cvar=0x557b3b61fa00, timeout=10000) > at ../../../nspr/pr/src/pthreads/ptsynch.c:387 > #3 0x00007f064526ef23 in eq_loop () > at /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b200) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fcd2a700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 17 (Thread 0x7f05fc529700 (LWP 11339)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040ad80) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fc529700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 16 (Thread 0x7f05fbd28700 (LWP 11340)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b7a0) > ---Type <return> to continue, or q <return> to quit--- > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fbd28700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 15 (Thread 0x7f05fb527700 (LWP 11341)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b8c0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fb527700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 14 (Thread 0x7f05fad26700 (LWP 11342)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040be60) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fad26700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 13 (Thread 0x7f05fa525700 (LWP 11343)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > ---Type <return> to continue, or q <return> to quit--- > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040bd40) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fa525700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 12 (Thread 0x7f05f9d24700 (LWP 11344)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040bc20) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f9d24700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 11 (Thread 0x7f05f9523700 (LWP 11345)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040bb00) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f9523700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 10 (Thread 0x7f05f8d22700 (LWP 11346)): > ---Type <return> to continue, or q <return> to quit--- > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b9e0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f8d22700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 9 (Thread 0x7f05f8521700 (LWP 11347)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a120) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f8521700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 8 (Thread 0x7f05f7d20700 (LWP 11348)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a240) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f7d20700) > at pthread_create.c:307 > ---Type <return> to continue, or q <return> to quit--- > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 7 (Thread 0x7f05f751f700 (LWP 11349)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a360) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f751f700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 6 (Thread 0x7f05f6d1e700 (LWP 11350)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a5a0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f6d1e700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 5 (Thread 0x7f05f651d700 (LWP 11351)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > ---Type <return> to continue, or q <return> to quit--- > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a480) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f651d700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 4 (Thread 0x7f05f5d1c700 (LWP 11352)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a7e0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f5d1c700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 3 (Thread 0x7f05f551b700 (LWP 11353)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a900) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f551b700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 2 (Thread 0x7f05f4d1a700 (LWP 11354)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=---Type <return> to continue, or q <return> to quit--- > 4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a6c0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f4d1a700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 1 (Thread 0x7f064594c940 (LWP 11328)): > #0 0x00007f064209ec3d in poll () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f0643059ba7 in poll (__timeout=250, __nfds=123, > __fds=0x557b404ae400) at /usr/include/bits/poll2.h:46 > #2 0x00007f0643059ba7 in _pr_poll_with_poll (pds=0x557b40bb8000, npds=123, > timeout=<optimized out>) > at ../../../nspr/pr/src/pthreads/ptio.c:4023 > #3 0x0000557b3a0ba149 in slapd_daemon () > #4 0x0000557b3a0abb15 in main () > (gdb) > (gdb) > > Deborah Crocker, PhD > Systems Engineer III > Office of Information Technology > The University of Alabama > Box 870346 > Tuscaloosa, AL 36587 > Office 205-348-3758 | Fax 205-348-9393 > [email protected] > > > -----Original Message----- > From: William Brown <[email protected]> > Sent: Sunday, May 31, 2020 8:00 PM > To: [email protected] > Subject: [EXTERNAL] [389-users] Re: Re: new server setup hanging > > > >> On 1 Jun 2020, at 10:54, Crocker, Deborah <[email protected]> wrote: >> >> We had to roll it back. There is one host running with it but the load is so >> light we never saw a problem. We think it was a known bug, maybe this: >> >> https://pagure.io/389-ds-base/issue/50329 > > That issue is fixed in 1.3.9, you are running 1.3.10, so that seems unlikely? > >> >> Do you want any info off the running host? > > As before, I'd need to see a gdb -p "pid" and `thread apply all bt` thanks. > When you gdb -p <pid>, it will tell you a command of what debug info packages > you need to install and how. You should install those before you run the gdb > commands. It will cause the server to "pause" when you attach gdb btw. > >> >> We'll now probably move into the 1.4.x trees. Any advice on which is the >> most stable? > > We try to make sure they are all stable - if you are using Red Hat/CentOS or > SLE/Suse Leap, then whatever 389-ds version are in those platforms is the > "best maintained" for that platform, and we'll resolve issues in them etc. I > think that's 1.4.2.x or 1.4.3.x from the top of my head at the moment for > RHEL 8.x and SLE 15.x. > >> >> -----Original Message----- >> From: William Brown <[email protected]> >> Sent: Sunday, May 31, 2020 6:25 PM >> To: [email protected] >> Subject: [EXTERNAL] [389-users] Re: new server setup hanging >> >> Hey there, >> >> >> We need to see the pstacks from all threads to really determine the cause >> here. Can you send us a complete read out? >> >> gdb -p "pid" >> >> thread apply all bt >> >> >> It'd be great if you can install debug info too to help. >> >> Thanks, >> >> >>> On 31 May 2020, at 05:09, Crocker, Deborah <[email protected]> wrote: >>> >>> Some more information from a coworker: >>> >>> Yeah, this sounds like an LDAP server bug. I haven't figured out what to >>> look at to pin it down, but when it's slow to connect, I can see with >>> strace that the primary thread hasn't called accept() yet for the >>> connection I'm trying to open. Once it does, the whole thing goes very >>> quickly, and I usually see a burst of other connections accepted and >>> handled at the same time. >>> >>> Deborah Crocker, PhD >>> Systems Engineer III >>> Office of Information Technology >>> The University of Alabama >>> Box 870346 >>> Tuscaloosa, AL 36587 >>> Office 205-348-3758 | Fax 205-348-9393 [email protected] >>> >>> >>> -----Original Message----- >>> From: Crocker, Deborah <[email protected]> >>> Sent: Saturday, May 30, 2020 2:08 PM >>> To: General discussion list for the 389 Directory server project. >>> <[email protected]> >>> Subject: [EXTERNAL] [389-users] new server setup hanging >>> >>> We're trying to move into our new server setup. We have one that seems to >>> be fine under a load but when we bring the next we're having trouble with >>> it hanging. The second does have more clients (and different) so there >>> could be something about what a client is doing. Here is the server: >>> 389-Directory/1.3.10.1 B2020.133.1625 Installed from EPEL, running on >>> CentOS Linux release 7.8.2003 >>> >>> And here is the pstack output listing the only thread that is not idle. Can >>> anyone tell me what is going on? >>> >>> Thread 44 (Thread 0x7f858e9b3700 (LWP 2515)): >>> #0 0x00007f860a90fe02 in slapi_atomic_load_32 () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #1 0x00007f860a8d4e8e in slapi_get_mapping_tree_node_by_dn () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #2 0x00007f860a8d5179 in slapi_be_select () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f860a9296a0 in vattr_test_filter () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #4 0x00007f860a8b6ec4 in slapi_vattr_filter_test_ext_internal () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #5 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #6 0x00007f8600a99e02 in acl__resource_match_aci () at >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so >>> #7 0x00007f8600a9b280 in acl_access_allowed () at >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so >>> #8 0x00007f8600aae9f7 in acl_access_allowed_main () at >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so >>> #9 0x00007f860a8f0cbc in plugin_call_acl_plugin () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #10 0x00007f860a8b638d in test_filter_access () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #11 0x00007f860a8b6fb5 in slapi_vattr_filter_test_ext_internal () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #12 0x00007f860a8b6d31 in slapi_vattr_filter_test_ext_internal () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #13 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #14 0x00007f85ff7c0df1 in ldbm_back_next_search_entry_ext () at >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f860a8deca6 in send_results_ext.constprop.5 () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #16 0x00007f860a8e0e09 in op_shared_search () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #17 0x0000557410dd3c0e in do_search () >>> #18 0x0000557410dc198a in connection_threadmain () >>> #19 0x00007f86086a0c5b in _pt_root () at /lib64/libnspr4.so >>> #20 0x00007f8608040ea5 in start_thread () at /lib64/libpthread.so.0 >>> #21 0x00007f86076ec8dd in clone () at /lib64/libc.so.6 >>> >>> Deborah Crocker, PhD >>> Systems Engineer III >>> Office of Information Technology >>> The University of Alabama >>> Box 870346 >>> Tuscaloosa, AL 36587 >>> Office 205-348-3758 | Fax 205-348-9393 [email protected] >>> >>> >>> -----Original Message----- >>> From: William Brown <[email protected]> >>> Sent: Wednesday, May 27, 2020 5:43 PM >>> To: [email protected] >>> Subject: [EXTERNAL] [389-users] Re: Re: Re: Advice to bring new >>> servers into production >>> >>> >>> >>>> On 27 May 2020, at 23:20, Crocker, Deborah <[email protected]> wrote: >>>> >>>> Thanks - I think we have enough ideas in here to get this going. One last >>>> question: >>>> If replication is set up through the host name - how often does the >>>> directory server do a DNS look up, or does it do it once on startup (or >>>> creation of the rep agreement)? >>> >>> I "think" it's every time it initiates the new connection - but remember, >>> for replication, that *is* quite different to a client doing a search, so >>> I'd be pretty careful about this. IMO you should be standing up your >>> replacement servers in parallel, joining them all, moving the IP's then >>> decomission the old servers. Alternately, you'll need an outage window to >>> shutdown your old servers, export the ldif, and then import and bring up >>> the new ones. >>> >>> I think having "IP's are a limited resource" really does make this >>> whole process much much harder than it needs to be for you ... :( >>> >>>> >>>> -----Original Message----- >>>> From: William Brown <[email protected]> >>>> Sent: Tuesday, May 26, 2020 10:48 PM >>>> To: [email protected] >>>> Subject: [EXTERNAL] [389-users] Re: Re: Advice to bring new servers >>>> into production >>>> >>>> There are a few options. The best would be a load balancer which has the >>>> ip's so that it's transparent to your LDAP servers where they are. >>>> >>>> But also as mentioned, the virtual IP's honestly is the best way. Linux >>>> can have multiple IP's on an interface so you can just have two IP's on >>>> one interface, andthat's the best way to do this. >>>> >>>> Alternately, don't rely on the IP, lower your DNS ttl's to a very short >>>> time, change the DNS A/AAAA records, and then do it that way. >>>> >>>> >>>> >>>>> On 27 May 2020, at 06:17, Crocker, Deborah <[email protected]> wrote: >>>>> >>>>> I'd like not to take up two ip addresses per host indefinitely. We have >>>>> re-IP'd our hosts before so I know we can to do this but it was during a >>>>> downtime when everything was restarted. Just trying to get away with not >>>>> restarting the masters. >>>>> >>>>> Deborah Crocker, PhD >>>>> Systems Engineer III >>>>> Office of Information Technology >>>>> The University of Alabama >>>>> Box 870346 >>>>> Tuscaloosa, AL 36587 >>>>> Office 205-348-3758 | Fax 205-348-9393 [email protected] >>>>> >>>>> From: Leo Pleiman <[email protected]> >>>>> Sent: Tuesday, May 26, 2020 3:08 PM >>>>> To: General discussion list for the 389 Directory server project. >>>>> <[email protected]> >>>>> Subject: [EXTERNAL] [389-users] Re: Advice to bring new servers >>>>> into production >>>>> >>>>> My experience has been that the replicas and consumers have a unique id, >>>>> more than just an IP address which creates the trust relationship with >>>>> the master. If your goal is to simply maintain an IP so your clients >>>>> don't have to be repointed, I would build each new LDAP host and >>>>> replication agreement, and then as you decommission the old hosts use >>>>> their IP address as a virtual IP address on the replacement host. It >>>>> would take a quick restart od the LDAP service to start a listener on the >>>>> virtual Ip address. >>>>> >>>>> >>>>> Leo Pleiman >>>>> Senior System Engineer >>>>> Direct 202-787-3622 >>>>> Cell 410-688-3873 >>>>> >>>>> >>>>> >>>>> On Tue, May 26, 2020 at 3:57 PM Crocker, Deborah <[email protected]> wrote: >>>>> We have a setup with 2 multi-masters and 3 consumers. We are now building >>>>> new host and want to put them in place ultimately at the same IP address >>>>> as the original ones. I need some advice on how to do this quickly and >>>>> cleanly. >>>>> >>>>> To add a new consumer the idea now is to set it up and set up >>>>> replications agreements from each master using consumer DNS name (don't >>>>> start continuous replication yet). After initializing new consumer from >>>>> one master - turn off old consumer, remove old consumer agreement from >>>>> each master, and re-IP new consumer. Do we need to restart masters to >>>>> re-read DNS or will it pick that up when it starts the next replication? >>>>> Is this the best way to do this? >>>>> >>>>> Thanks >>>>> >>>>> Deborah Crocker, PhD >>>>> Systems Engineer III >>>>> Office of Information Technology >>>>> The University of Alabama >>>>> Box 870346 >>>>> Tuscaloosa, AL 36587 >>>>> Office 205-348-3758 | Fax 205-348-9393 [email protected] >>>>> >>>>> _______________________________________________ >>>>> 389-users mailing list -- [email protected] To >>>>> unsubscribe send an email to >>>>> [email protected] >>>>> Fedora Code of Conduct: >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: >>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: >>>>> https://lists.fedoraproject.org/archives/list/[email protected] >>>>> a p r oject.org _______________________________________________ >>>>> 389-users mailing list -- [email protected] To >>>>> unsubscribe send an email to >>>>> [email protected] >>>>> Fedora Code of Conduct: >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: >>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: >>>>> https://lists.fedoraproject.org/archives/list/[email protected] >>>>> a >>>>> p >>>>> r >>>>> oject.org >>>> >>>> - >>>> Sincerely, >>>> >>>> William Brown >>>> >>>> Senior Software Engineer, 389 Directory Server SUSE Labs >>>> _______________________________________________ >>>> 389-users mailing list -- [email protected] To >>>> unsubscribe send an email to [email protected] >>>> Fedora Code of Conduct: >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: >>>> https://lists.fedoraproject.org/archives/list/[email protected] >>>> p r oject.org _______________________________________________ >>>> 389-users mailing list -- [email protected] To >>>> unsubscribe send an email to [email protected] >>>> Fedora Code of Conduct: >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: >>>> https://lists.fedoraproject.org/archives/list/[email protected] >>>> p >>>> r >>>> oject.org >>> >>> - >>> Sincerely, >>> >>> William Brown >>> >>> Senior Software Engineer, 389 Directory Server SUSE Labs >>> _______________________________________________ >>> 389-users mailing list -- [email protected] To >>> unsubscribe send an email to [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/[email protected] >>> r oject.org _______________________________________________ >>> 389-users mailing list -- [email protected] To >>> unsubscribe send an email to [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/[email protected] >>> r oject.org _______________________________________________ >>> 389-users mailing list -- [email protected] To >>> unsubscribe send an email to [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/[email protected] >>> r >>> oject.org >> >> - >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server SUSE Labs >> _______________________________________________ >> 389-users mailing list -- [email protected] To >> unsubscribe send an email to [email protected] >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/[email protected] >> oject.org _______________________________________________ >> 389-users mailing list -- [email protected] To >> unsubscribe send an email to [email protected] >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/[email protected] >> oject.org > > - > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server SUSE Labs > _______________________________________________ > 389-users mailing list -- [email protected] To unsubscribe > send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > _______________________________________________ > 389-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] - Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
_______________________________________________ 389-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
