Hey there,
We need to see the pstacks from all threads to really determine the cause here. Can you send us a complete read out? gdb -p "pid" thread apply all bt It'd be great if you can install debug info too to help. Thanks, > On 31 May 2020, at 05:09, Crocker, Deborah <[email protected]> wrote: > > Some more information from a coworker: > > Yeah, this sounds like an LDAP server bug. I haven't figured out what to look > at to pin it down, but when it's slow to connect, I can see with strace that > the primary thread hasn't called accept() yet for the connection I'm trying > to open. Once it does, the whole thing goes very quickly, and I usually see a > burst of other connections accepted and handled at the same time. > > Deborah Crocker, PhD > Systems Engineer III > Office of Information Technology > The University of Alabama > Box 870346 > Tuscaloosa, AL 36587 > Office 205-348-3758 | Fax 205-348-9393 > [email protected] > > > -----Original Message----- > From: Crocker, Deborah <[email protected]> > Sent: Saturday, May 30, 2020 2:08 PM > To: General discussion list for the 389 Directory server project. > <[email protected]> > Subject: [EXTERNAL] [389-users] new server setup hanging > > We're trying to move into our new server setup. We have one that seems to be > fine under a load but when we bring the next we're having trouble with it > hanging. The second does have more clients (and different) so there could be > something about what a client is doing. Here is the server: > 389-Directory/1.3.10.1 B2020.133.1625 > Installed from EPEL, running on > CentOS Linux release 7.8.2003 > > And here is the pstack output listing the only thread that is not idle. Can > anyone tell me what is going on? > > Thread 44 (Thread 0x7f858e9b3700 (LWP 2515)): > #0 0x00007f860a90fe02 in slapi_atomic_load_32 () at > /usr/lib64/dirsrv/libslapd.so.0 > #1 0x00007f860a8d4e8e in slapi_get_mapping_tree_node_by_dn () at > /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f860a8d5179 in slapi_be_select () at > /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f860a9296a0 in vattr_test_filter () at > /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f860a8b6ec4 in slapi_vattr_filter_test_ext_internal () at > /usr/lib64/dirsrv/libslapd.so.0 > #5 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at > /usr/lib64/dirsrv/libslapd.so.0 > #6 0x00007f8600a99e02 in acl__resource_match_aci () at > /usr/lib64/dirsrv/plugins/libacl-plugin.so > #7 0x00007f8600a9b280 in acl_access_allowed () at > /usr/lib64/dirsrv/plugins/libacl-plugin.so > #8 0x00007f8600aae9f7 in acl_access_allowed_main () at > /usr/lib64/dirsrv/plugins/libacl-plugin.so > #9 0x00007f860a8f0cbc in plugin_call_acl_plugin () at > /usr/lib64/dirsrv/libslapd.so.0 > #10 0x00007f860a8b638d in test_filter_access () at > /usr/lib64/dirsrv/libslapd.so.0 > #11 0x00007f860a8b6fb5 in slapi_vattr_filter_test_ext_internal () at > /usr/lib64/dirsrv/libslapd.so.0 > #12 0x00007f860a8b6d31 in slapi_vattr_filter_test_ext_internal () at > /usr/lib64/dirsrv/libslapd.so.0 > #13 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at > /usr/lib64/dirsrv/libslapd.so.0 > #14 0x00007f85ff7c0df1 in ldbm_back_next_search_entry_ext () at > /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f860a8deca6 in send_results_ext.constprop.5 () at > /usr/lib64/dirsrv/libslapd.so.0 > #16 0x00007f860a8e0e09 in op_shared_search () at > /usr/lib64/dirsrv/libslapd.so.0 > #17 0x0000557410dd3c0e in do_search () > #18 0x0000557410dc198a in connection_threadmain () > #19 0x00007f86086a0c5b in _pt_root () at /lib64/libnspr4.so > #20 0x00007f8608040ea5 in start_thread () at /lib64/libpthread.so.0 > #21 0x00007f86076ec8dd in clone () at /lib64/libc.so.6 > > Deborah Crocker, PhD > Systems Engineer III > Office of Information Technology > The University of Alabama > Box 870346 > Tuscaloosa, AL 36587 > Office 205-348-3758 | Fax 205-348-9393 > [email protected] > > > -----Original Message----- > From: William Brown <[email protected]> > Sent: Wednesday, May 27, 2020 5:43 PM > To: [email protected] > Subject: [EXTERNAL] [389-users] Re: Re: Re: Advice to bring new servers into > production > > > >> On 27 May 2020, at 23:20, Crocker, Deborah <[email protected]> wrote: >> >> Thanks - I think we have enough ideas in here to get this going. One last >> question: >> If replication is set up through the host name - how often does the >> directory server do a DNS look up, or does it do it once on startup (or >> creation of the rep agreement)? > > I "think" it's every time it initiates the new connection - but remember, for > replication, that *is* quite different to a client doing a search, so I'd be > pretty careful about this. IMO you should be standing up your replacement > servers in parallel, joining them all, moving the IP's then decomission the > old servers. Alternately, you'll need an outage window to shutdown your old > servers, export the ldif, and then import and bring up the new ones. > > I think having "IP's are a limited resource" really does make this whole > process much much harder than it needs to be for you ... :( > >> >> -----Original Message----- >> From: William Brown <[email protected]> >> Sent: Tuesday, May 26, 2020 10:48 PM >> To: [email protected] >> Subject: [EXTERNAL] [389-users] Re: Re: Advice to bring new servers >> into production >> >> There are a few options. The best would be a load balancer which has the >> ip's so that it's transparent to your LDAP servers where they are. >> >> But also as mentioned, the virtual IP's honestly is the best way. Linux can >> have multiple IP's on an interface so you can just have two IP's on one >> interface, andthat's the best way to do this. >> >> Alternately, don't rely on the IP, lower your DNS ttl's to a very short >> time, change the DNS A/AAAA records, and then do it that way. >> >> >> >>> On 27 May 2020, at 06:17, Crocker, Deborah <[email protected]> wrote: >>> >>> I’d like not to take up two ip addresses per host indefinitely. We have >>> re-IP’d our hosts before so I know we can to do this but it was during a >>> downtime when everything was restarted. Just trying to get away with not >>> restarting the masters. >>> >>> Deborah Crocker, PhD >>> Systems Engineer III >>> Office of Information Technology >>> The University of Alabama >>> Box 870346 >>> Tuscaloosa, AL 36587 >>> Office 205-348-3758 | Fax 205-348-9393 [email protected] >>> >>> From: Leo Pleiman <[email protected]> >>> Sent: Tuesday, May 26, 2020 3:08 PM >>> To: General discussion list for the 389 Directory server project. >>> <[email protected]> >>> Subject: [EXTERNAL] [389-users] Re: Advice to bring new servers into >>> production >>> >>> My experience has been that the replicas and consumers have a unique id, >>> more than just an IP address which creates the trust relationship with the >>> master. If your goal is to simply maintain an IP so your clients don't have >>> to be repointed, I would build each new LDAP host and replication >>> agreement, and then as you decommission the old hosts use their IP address >>> as a virtual IP address on the replacement host. It would take a quick >>> restart od the LDAP service to start a listener on the virtual Ip address. >>> >>> >>> Leo Pleiman >>> Senior System Engineer >>> Direct 202-787-3622 >>> Cell 410-688-3873 >>> >>> >>> >>> On Tue, May 26, 2020 at 3:57 PM Crocker, Deborah <[email protected]> wrote: >>> We have a setup with 2 multi-masters and 3 consumers. We are now building >>> new host and want to put them in place ultimately at the same IP address as >>> the original ones. I need some advice on how to do this quickly and cleanly. >>> >>> To add a new consumer the idea now is to set it up and set up replications >>> agreements from each master using consumer DNS name (don't start continuous >>> replication yet). After initializing new consumer from one master - turn >>> off old consumer, remove old consumer agreement from each master, and re-IP >>> new consumer. Do we need to restart masters to re-read DNS or will it pick >>> that up when it starts the next replication? Is this the best way to do >>> this? >>> >>> Thanks >>> >>> Deborah Crocker, PhD >>> Systems Engineer III >>> Office of Information Technology >>> The University of Alabama >>> Box 870346 >>> Tuscaloosa, AL 36587 >>> Office 205-348-3758 | Fax 205-348-9393 [email protected] >>> >>> _______________________________________________ >>> 389-users mailing list -- [email protected] To >>> unsubscribe send an email to [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/[email protected] >>> r oject.org _______________________________________________ >>> 389-users mailing list -- [email protected] To >>> unsubscribe send an email to [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/[email protected] >>> r >>> oject.org >> >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server SUSE Labs >> _______________________________________________ >> 389-users mailing list -- [email protected] To >> unsubscribe send an email to [email protected] >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/[email protected] >> oject.org _______________________________________________ >> 389-users mailing list -- [email protected] To >> unsubscribe send an email to [email protected] >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/[email protected] >> oject.org > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server SUSE Labs > _______________________________________________ > 389-users mailing list -- [email protected] To unsubscribe > send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > _______________________________________________ > 389-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > _______________________________________________ > 389-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
