Hi Grant, Good finding about the root cause !
About the solution, I think we should continue to send an anonymous bind but considers that ping is successful if the return code is either LDAP_SUCCESS or LDAP_INAPPROPRIATE_AUTH (anyway we do not care that the bind is really successful, we only want to know that server is responsive) FYI: My concerns with the search are that: - we would hit the same issue if acl deny the search - it would increase the root dse contention Regards Pierre On Mon, Mar 8, 2021 at 5:56 AM Grant Byers <[email protected]> wrote: > Confirmed. I made the following simple change and it allows cb_ping_farm > to work with anonymous binds only enabled for rootdse; > > > diff -urN a/ldap/servers/plugins/chainingdb/cb_conn_stateless.c > b/ldap/servers/plugins/chainingdb/cb_conn_stateless.c > --- a/ldap/servers/plugins/chainingdb/cb_conn_stateless.c 2020-03-17 > 04:52:57.000000000 +1000 > +++ b/ldap/servers/plugins/chainingdb/cb_conn_stateless.c 2021-03-08 > 14:04:48.413647052 +1000 > @@ -883,7 +883,7 @@ > /* NOTE: This will fail if we implement the ability to disable > anonymous bind */ > - rc = ldap_search_ext_s(ld, NULL, LDAP_SCOPE_BASE, "objectclass=*", > attrs, 1, NULL, > + rc = ldap_search_ext_s(ld, "", LDAP_SCOPE_BASE, "objectclass=*", > attrs, 1, NULL, > NULL, &timeout, 1, &result); > if (LDAP_SUCCESS != rc) { > slapi_ldap_unbind(ld); > > > I don't believe this will break any functionality, but since we're running > RHEL7, i'll raise this with Red Hat directly and they can review and/or > push upstream. > > Regards, > Grant > > ------------------------------ > *From:* Grant Byers <[email protected]> > *Sent:* Monday, March 8, 2021 12:27 PM > *To:* [email protected] <[email protected] > > > *Subject:* Re: [389-users] Re: Chain on Update problem > > Thanks. > > I have tested various combinations of the tuning params without success. > I've done further debugging and confirmed that it always starts after a > bind operation timeout. Looking into the chaining plugin code, I see that > on operation timeout results in a call to cb_ping_farm to see if we can > find another server in the pool that is available. However, it performs > this operation (the comment is telling); > > /* NOTE: This will fail if we implement the ability to disable > anonymous bind */ > rc = ldap_search_ext_s(ld, NULL, LDAP_SCOPE_BASE, "objectclass=*", > attrs, 1, NULL, > NULL, &timeout, 1, &result); > if (LDAP_SUCCESS != rc) { > slapi_ldap_unbind(ld); > cb_update_failed_conn_cpt(cb); > return LDAP_SERVER_DOWN; > } > > So basically, because we've disallowed anonymous bind for anything but > rootdse, it will always fail to find another available server. I have > confirmed this by allowing anonymous bind on our masters while the issue > was present, then subsequent binds on the consumers start working again. > > I would think it more appropriate for that code to do a search against the > rootdse instead. Is there any good reason why it shouldn't? If not, I might > test modifying it. > > > Thanks, > Grant > > > ------------------------------ > *From:* William Brown <[email protected]> > *Sent:* Friday, March 5, 2021 3:52 PM > *To:* [email protected] <[email protected] > > > *Subject:* [389-users] Re: Chain on Update problem > > [External Mail] > > > > On 5 Mar 2021, at 12:03, Grant Byers <[email protected]> wrote: > > > > Hi All, > > > > Version: 1.3.10 > > > > In our environment, we'd like to use a chaining backend to push BIND > operations up to masters by way of the consumer (rather than client > referral). We'd like to do this to ensure password lockout attributes are > propagated to all consumers equally via our standard replication > agreements. This is described here - > https://directory.fedoraproject.org/docs/389ds/howto/howto-chainonupdate.html > . > > > > NOTE, we do not have hubs in our topology. Just masters and consumers, > so no intermediate chaining. > > > > We tested this process in our environment and it worked beautifully > until we took it to production. Currently, we have just 2 masters and they > are both sitting on some over-subscribed hardware that suffers from I/O > starvation at certain times of the day. The plan is to scale out our > masters eventually, but we're a little hamstrung with other projects and > priorities. It worked extremely well until that time of day when masters > suffered from I/O starvation, and hence, very long I/O wait times. This is > generally short lived and happens at alternate times of the day for each of > the masters. However, it seems that once both nsfarmservers have "failed", > there is never any attempt by the consumer to retry them. This leads to > bind errors as follows; > > > > ldapwhoami -x -D "<binddn>" -W > > Enter LDAP Password: > > ldap_bind: Operations error (1) > > additional info: FARM SERVER TEMPORARY UNAVAILABLE > > > > Except it is not temporary. It never recovers, even though all members > of nsfarmservers are now healthy again (and are never unhealthy at the same > time). We can confirm this by performing binds from the consumers directly > against the masters. I thought that setting nsConnectionLife to something > larger than 0 (indefinite) would help this, but it has not. > > The chain on update appears to use the chaining plugin timeouts, so you > could look at adjusting these parameters which may help. > > nsBindTimeout > nsOperationTimeout > nsBindRetryLimit > nsMaxResponseDelay > nsMaxTestResponseDelay > > > > > > > Is this by design, a bug, or an implementation fault on my behalf? > Configuration below; > > > > Thanks, > > Grant > > > > > > > > ## On masters, create a dedicated user for chaining backend > > dn: cn=proxy,cn=config > > objectClass: person > > objectClass: top > > cn: proxy > > sn: admin > > > > ## On all consumers, create chaining backend; > > dn: cn=chainbe1,cn=chaining database,cn=plugins,cn=config > > objectclass: top > > objectclass: extensibleObject > > objectclass: nsBackendInstance > > nsslapd-suffix: <suffix> > > nsfarmserverurl: ldaps://<master1>:636 <master2>:636/ > > nsMultiplexorBindDN: <binddn>> > > nsMultiplexorCredentials: <bindpw> > > nsCheckLocalACI: on > > nsConnectionLife: 30 > > cn: chainbe1 > > > > ## On all consumers, add the backend and repl_chain_on_update function > > dn: cn="<suffix>",cn=mapping tree,cn=config > > changetype: modify > > add: nsslapd-backend > > nsslapd-backend: chainbe1 > > - > > add: nsslapd-distribution-plugin > > nsslapd-distribution-plugin: libreplication-plugin > > - > > add: nsslapd-distribution-funct > > nsslapd-distribution-funct: repl_chain_on_update > > > > ## On all servers, enable global pasword policy > > dn: cn=config > > changetype: modify > > replace: passwordIsGlobalPolicy > > passwordIsGlobalPolicy: on > > > > _______________________________________________ > > 389-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs, Australia > _______________________________________________ > 389-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > _______________________________________________ > 389-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > -- -- 389 Directory Server Development Team
_______________________________________________ 389-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
