Sean Hogan wrote:
Hello All,
Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50. I also think
(not sure on this yet) that they changed ntp.. ntp used to point at my
ipas.. but they look like they are now pointing elsewhere. Everything
was stable at 6.7 3.0.47 pointing to IPA for NTP. However.. they all
seem to have the same date.
My master first IPA is acting up. Replication is off, kerberos seems to
be off, DNS is off and I think IPA in general on it is toast.
We do have 8 IPAs.. only FirstMaster is acting up it seems right now and
all either running on KVM or ESXI.
[God@FirstMasterIPA slapd-DOMAIN-LOCAL]# kinit admin
kinit: Generic error (see e-text) while getting initial credential
ipactl status should show what services are running. It looks like the
KDC is responding but can't talk to the LDAP backend.
slapd-DOMAIN-LOCAL
[01/Jun/2016:18:25:43 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure. Minor code may provide more information (Cannot contact any
KDC for realm 'DOMAIN.LOCAL')) errno 115 (Operation now in progress)
[01/Jun/2016:18:25:43 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
agmt="cn=meToipaserv2.domain.local" (ipaserv2:389): Replication bind
with GSSAPI auth resumed
[01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
agmt="cn=meToipaserv3.domain.local" (ipaserv3:389): Replication bind
with GSSAPI auth resumed
[01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
agmt="cn=meToipaserv4.domain.local" (ipaserv4:389): Replication bind
with GSSAPI auth resumed
[01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin -
agmt="cn=meToipaserv5.domain.local" (ipaserv5:389): Replication bind
with GSSAPI auth resumed
[01/Jun/2016:18:28:04 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
Failure: gss_accept_sec_context) errno 0 (Success)
[01/Jun/2016:18:28:04 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
[01/Jun/2016:18:28:13 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure. Minor code may provide more information (No credentials
cache found)) errno 2 (No such file or directory)
[01/Jun/2016:18:28:13 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[01/Jun/2016:18:33:03 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
Failure: gss_accept_sec_context) errno 0 (Success)
[01/Jun/2016:18:33:03 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
[01/Jun/2016:18:33:18 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure. Minor code may provide more information (No credentials
cache found)) errno 2 (No such file or directory)
[01/Jun/2016:18:33:18 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[01/Jun/2016:18:38:03 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI
Failure: gss_accept_sec_context) errno 0 (Success)
And this makes it look like it can't talk to the KDC.
I'd check for SELinux errors, ausearch -m AVC -ts recent
I think the rest is just indication that something is wrong with either
the LDAP servers, the KDC or both.
You may also want to look at /var/log/ipaupgrade.log to ensure that the
upgrade was successful.
rob
[God@FirstMasterIPA slapd-DOMAIN-LOCAL]# ipa-replica-manage -v list
--------------> just hangs and never returns
[God@FirstMasterIPA slapd-DOMAIN-LOCAL]# ipactl start ------------->Just
hangs here as well.. never gets to the KDC.
Starting Directory Service
Starting dirsrv:
PKI-IPA... already running [ OK ]
DOMAIN-LOCAL... already running [ OK ]
If I run nslookup it fails over to a Replica for the DNS resolution
instead of resolving ips itself.
PKI log shows a bunch of this:
[02/Jun/2016:11:15:25 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca"
(ipaserver2:7389): Replication bind with SIMPLE auth failed: LDAP error
-1 (Can't contact LDAP server) ((null))
[02/Jun/2016:11:15:34 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca"
(ipaserver2:7389): Replication bind with SIMPLE auth resumed
[02/Jun/2016:11:16:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:16:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:11:21:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:22:06 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:11:26:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:26:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:11:31:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:31:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:11:36:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:36:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:11:41:46 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:41:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:11:45:16 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:45:16 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
(ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error
-1 (Can't contact LDAP server) ((null))
[02/Jun/2016:11:45:25 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
(ipaserver3:7389): Replication bind with SIMPLE auth resumed
[02/Jun/2016:11:46:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:46:56 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:11:51:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:51:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:11:56:46 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:11:56:51 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:12:01:36 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:12:01:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:12:05:33 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:12:05:33 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
(ipaserver3:7389): Replication bind with SIMPLE auth failed: LDAP error
-1 (Can't contact LDAP server) ((null))
[02/Jun/2016:12:06:01 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
[02/Jun/2016:12:06:06 -0400] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ipaserver3.domain.local-pki-ca"
(ipaserver3:7389): Replication bind with SIMPLE auth resumed
[02/Jun/2016:12:06:31 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[02/Jun/2016:12:06:41 -0400] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success)
NTP seems OK
[God@FirstMasterIPA slapd-PKI-IPA]# date
Thu Jun 2 12:23:00 EDT 2016
[God@ipaserver3 ~]# date
Thu Jun 2 12:23:02 EDT 2016
Sean Hogan
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
