So I changed the hostnames in krb5.conf

[realms]
 <DOMAIN> = {
  kdc = <MASTERHOST>:88
  master_kdc = <MASTERHOST>:88
  admin_server = <MASTERHOST>:749
  default_domain = <DOMAIN>
  pkinit_anchors = FILE:/etc/ipa/ca.crt
}

Service still will not start however now in the access log instead of showing 
the connection from master to replica it shows replica to replica

[10/Nov/2015:08:51:03 -0500] conn=52 op=3 UNBIND
[10/Nov/2015:08:51:03 -0500] conn=52 op=3 fd=64 closed - U1
[10/Nov/2015:08:51:03 -0500] conn=52 op=0 RESULT err=14 tag=97 nentries=0 
etime=0, SASL bind in progress
[10/Nov/2015:08:51:05 -0500] conn=53 fd=64 slot=64 connection from <REPLICA_IP> 
to <REPLICA_IP>
[10/Nov/2015:08:51:05 -0500] conn=53 op=0 BIND dn="" method=sasl version=3 
mech=GSSAPI
[10/Nov/2015:08:51:05 -0500] conn=53 op=1 BIND dn="" method=sasl version=3 
mech=GSSAPI
[10/Nov/2015:08:51:05 -0500] conn=53 op=1 RESULT err=14 tag=97 nentries=0 
etime=0, SASL bind in progress
[10/Nov/2015:08:51:05 -0500] conn=53 op=2 BIND dn="" method=sasl version=3 
mech=GSSAPI
[10/Nov/2015:08:51:05 -0500] conn=53 op=2 RESULT err=49 tag=97 nentries=0 
etime=0
[10/Nov/2015:08:51:05 -0500] conn=53 op=3 UNBIND
[10/Nov/2015:08:51:05 -0500] conn=53 op=3 fd=64 closed - U1
[10/Nov/2015:08:51:05 -0500] conn=53 op=0 RESULT err=14 tag=97 nentries=0 
etime=0, SASL bind in progress

-----Original Message-----
From: Alexander Bokovoy [mailto:aboko...@redhat.com] 
Sent: Tuesday, November 10, 2015 8:41 AM
To: Gronde, Christopher (Contractor) <christopher.gro...@fincen.gov>
Cc: Rob Crittenden <rcrit...@redhat.com>; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos authentication 
error)

On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:
>Where can I verify or change the credentials it is trying to use?  Is it my 
>LDAP password?
No, according to your logs, it is your LDAP master trying to replicate (push 
changes) to your LDAP replica:
>>[09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection from 
>><MASTER_IP> to <REPLICA_IP>
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl 
>>version=3 mech=GSSAPI

If that is true, it would be ldap/<master> Kerberos principal talking to 
ldap/<replica> Kerberos principal. If that fails, it means master and replica 
KDCs have different understanding of both ldap/<master> and ldap/<replica> keys 
which most likely means keys were rotated on master and weren't propagated to 
replica.

How to solve it? One possibility is to set master's hostname as KDC address in 
krb5.conf on replica, forcing LDAP server on replica to use master's KDC. I'm 
absolutely not sure this will actually work but at least it allows to see if we 
are indeed dealing with inconsistent state of service principals' keys.

>
>-----Original Message-----
>From: Alexander Bokovoy [mailto:aboko...@redhat.com]
>Sent: Tuesday, November 10, 2015 8:18 AM
>To: Gronde, Christopher (Contractor) <christopher.gro...@fincen.gov>
>Cc: Rob Crittenden <rcrit...@redhat.com>; freeipa-users@redhat.com
>Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>authentication error)
>
>On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:
>>When I tried to start the service again I got no response from tail of 
>>the log, but this is a repeating entry I see in the access log
>>
>>[09/Nov/2015:15:01:04 -0500] conn=1 fd=64 slot=64 connection from
>>127.0.0.1 to 127.0.0.1
>>[09/Nov/2015:15:01:04 -0500] conn=1 op=-1 fd=64 closed - B1
>>[09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection from 
>><MASTER_IP> to <REPLICA_IP>
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
>>version=3 mech=GSSAPI
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=0 RESULT err=14 tag=97
>>nentries=0 etime=0, SASL bind in progress
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=1 BIND dn="" method=sasl
>>version=3 mech=GSSAPI
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=1 RESULT err=14 tag=97
>>nentries=0 etime=0, SASL bind in progress
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=2 BIND dn="" method=sasl
>>version=3 mech=GSSAPI
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=2 RESULT err=49 tag=97
>>nentries=0 etime=0
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=3 UNBIND
>>[09/Nov/2015:15:02:01 -0500] conn=2 op=3 fd=64 closed - U1
>>
>>Does anyone know what err=14 or err=49 are?
>err=14 means SASL bind in progress -- i.e. multi-round processing is ongoing. 
>This is normal for SASL GSSAPI.
>
>err=49 is wrong password or username, i.e. credentials were incorrect.
>It may also mean that LDAP server side was unable to process Kerberos 
>negotiation due to not having a current Kerberos ticket for own service
>(LDAP) and trying to request it from the Kerberos KDC but Kerberos KDC is down.
>
>>
>>-----Original Message-----
>>From: Rob Crittenden [mailto:rcrit...@redhat.com]
>>Sent: Monday, November 09, 2015 3:26 PM
>>To: Gronde, Christopher (Contractor) <christopher.gro...@fincen.gov>; 
>>Alexander Bokovoy <aboko...@redhat.com>
>>Cc: freeipa-users@redhat.com
>>Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>authentication error)
>>
>>Gronde, Christopher (Contractor) wrote:
>>> Nothing bad came back and there is definitely data in the tree.
>>
>>Ok, I guess I'd try to start the kdc again and then watch the 389-ds access 
>>log (buffered) to:
>>
>>1. See if it is binding at all
>>2. See what the search is and what, if any, results were returned
>>
>>This would be in /var/log/dirsrv/slapd-YOUR_REALM/access
>>
>>rob
>>
>>>
>>> -----Original Message-----
>>> From: Rob Crittenden [mailto:rcrit...@redhat.com]
>>> Sent: Monday, November 09, 2015 11:46 AM
>>> To: Gronde, Christopher (Contractor) 
>>> <christopher.gro...@fincen.gov>; Alexander Bokovoy 
>>> <aboko...@redhat.com>
>>> Cc: freeipa-users@redhat.com
>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>> authentication error)
>>>
>>> Gronde, Christopher (Contractor) wrote:
>>>> I restarted dirsrv and attempted to start krb5kdc and this is what 
>>>> the error log shows
>>>>
>>>> # tail /var/log/dirsrv/slapd-ITMODEV-GOV/errors
>>>> [09/Nov/2015:11:01:02 -0500] - WARNING: userRoot: entry cache size 
>>>> 10485760B is less than db size 28016640B; We recommend to increase the 
>>>> entry cache size nsslapd-cachememsize.
>>>> [09/Nov/2015:11:01:02 -0500] - slapd started.  Listening on All 
>>>> Interfaces port 389 for LDAP requests
>>>> [09/Nov/2015:11:06:04 -0500] - slapd shutting down - signaling 
>>>> operation threads
>>>> [09/Nov/2015:11:06:04 -0500] - slapd shutting down - closing down 
>>>> internal subsystems and plugins
>>>> [09/Nov/2015:11:06:04 -0500] - Waiting for 4 database threads to 
>>>> stop
>>>> [09/Nov/2015:11:06:04 -0500] - All database threads now stopped
>>>> [09/Nov/2015:11:06:04 -0500] - slapd stopped.
>>>> [09/Nov/2015:11:14:20 -0500] - 389-Directory/1.2.11.15
>>>> B2015.247.1737 starting up
>>>> [09/Nov/2015:11:14:20 -0500] - WARNING: userRoot: entry cache size 
>>>> 10485760B is less than db size 28016640B; We recommend to increase the 
>>>> entry cache size nsslapd-cachememsize.
>>>> [09/Nov/2015:11:14:20 -0500] - slapd started.  Listening on All 
>>>> Interfaces port 389 for LDAP requests
>>>
>>> Ok, that's good.
>>>
>>> I'd do something like this to see what is in the db (substitute example.com 
>>> with your domain):
>>>
>>> $ ldapsearch -x -D 'cn=Directory Manager' -W -s one -b 
>>> cn=kerberos,dc=example,dc=com
>>>
>>> (don't post the output as it would include the kerberos master key).
>>>
>>> If that returns nothing that's bad.
>>>
>>> If it succeeds I'd broaden the search base a bit to see what data 
>>> you do
>>> have:
>>>
>>> $ ldapsearch -x -D 'cn=Directory Manager' -W -b 
>>> cn=groups,cn=accounts,dc=example,dc=com
>>>
>>> I picked groups because usually groups << users in numbers. This is just to 
>>> see if you have data in the tree.
>>>
>>> Let us know if either or both turns up nothing.
>>>
>>> rob
>>>
>>>>
>>>> -----Original Message-----
>>>> From: Alexander Bokovoy [mailto:aboko...@redhat.com]
>>>> Sent: Monday, November 09, 2015 10:51 AM
>>>> To: Gronde, Christopher (Contractor) 
>>>> <christopher.gro...@fincen.gov>
>>>> Cc: freeipa-users@redhat.com
>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos 
>>>> authentication error)
>>>>
>>>> On Mon, 09 Nov 2015, Gronde, Christopher (Contractor) wrote:
>>>>> Hello all!
>>>>>
>>>>> On my replica IPA server after fixing a cert issue that had been going on 
>>>>> for sometime, I have all my certs figured out but the krb5kdc service 
>>>>> will not start.
>>>>>
>>>>> # service krb5kdc start
>>>>> Starting Kerberos 5 KDC: krb5kdc: cannot initialize realm ITMODEV.GOV - 
>>>>> see log file for details                  [FAILED]
>>>>>
>>>>> # cat /var/log/krb5kdc.log
>>>>> krb5kdc: Server error - while fetching master key K/M for realm 
>>>>> ITMODEV.GOV
>>>>> krb5kdc: Server error - while fetching master key K/M for realm 
>>>>> ITMODEV.GOV
>>>>> krb5kdc: Server error - while fetching master key K/M for realm 
>>>>> ITMODEV.GOV
>>>>>
>>>>> I found this article online:
>>>>> http://research.imb.uq.edu.au/~l.rathbone/ldap/kerberos.shtml
>>>>>
>>>>> Which stated it might be because The slave KDC does not have a 
>>>>> stash file (.k5.EXAMPLE.COM). You need to create one.  Tried the 
>>>>> command
>>>>> listed:
>>>>>
>>>>> # kdb5_util stash
>>>>> kdb5_util: Server error while retrieving master entry
>>>>>
>>>>> No further information found on the proceeding error above for the 
>>>>> kdb5_util command.
>>>>>
>>>>> Any thoughts?
>>>> First: don't use instructions which are not related to IPA, please.
>>>>
>>>> FreeIPA has its own LDAP driver for KDC and instructions for anything else 
>>>> do not apply here at all.
>>>>
>>>> If you see 'Server error - while fetching master key ..' it means KDC LDAP 
>>>> driver was unable to contact LDAP server. Does LDAP server work on the 
>>>> replica? What is in its error log 
>>>> (/var/log/dirsrv/slapd-ITMODEV-GOV/errors)?
>>>>
>>>> --
>>>> / Alexander Bokovoy
>>>>
>>>>
>>>
>>>
>>
>>
>
>--
>/ Alexander Bokovoy
>

--
/ Alexander Bokovoy


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to