On 11/10/2015 09:49 AM, Gronde, Christopher (Contractor) wrote:
Note comipa01 is the master and comipa02 is the replica that is having the KDC 
issue

# ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b 
"dc=itmodev,dc=gov" '(krbprincipalname=ldap/comipa01.itmodev.gov*)'
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=itmodev,dc=gov> with scope subtree
# filter: (krbprincipalname=ldap/comipa01.itmodev.gov*)
# requesting: ALL
#

# ldap/comipa01.itmodev....@itmodev.gov, services, accounts, itmodev.gov
dn: krbprincipalname=ldap/comipa01.itmodev....@itmodev.gov,cn=services,cn=acco
  unts,dc=itmodev,dc=gov
krbLastSuccessfulAuth: 20151015230212Z
ipaKrbPrincipalAlias: ldap/comipa01.itmodev....@itmodev.gov
krbExtraData:: AAL6CaBOYWRtaW4vYWRtaW5ASVRNT0RFVi5HT1YA
krbExtraData:: AAgBAA==
userCertificate:: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXREDACTEDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
krbPasswordExpiration: 19700101000000Z
ipaUniqueID: 15a897e8-fb11-11e0-b8e5-001a4a106404
krbLastPwdChange: 20111020114602Z
krbPrincipalName: ldap/comipa01.itmodev....@itmodev.gov
krbLoginFailedCount: 0
managedBy: fqdn=comipa01.itmodev.gov,cn=computers,cn=accounts,dc=itmodev,dc=go
  v
objectClass: ipaobject
objectClass: top
objectClass: ipaservice
objectClass: pkiuser
objectClass: krbprincipal
objectClass: krbprincipalaux
objectClass: krbTicketPolicyAux
objectClass: ipakrbprincipal
krbPrincipalKey:: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXREDACTEDXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root@comipa02 ~]#

# ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b 
"dc=itmodev,dc=gov" '(krbprincipalname=ldap/comipa02.itmodev.gov*)'
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=itmodev,dc=gov> with scope subtree
# filter: (krbprincipalname=ldap/comipa02.itmodev.gov*)
# requesting: ALL
#

# ldap/comipa02.itmodev....@itmodev.gov, services, accounts, itmodev.gov
dn: krbprincipalname=ldap/comipa02.itmodev....@itmodev.gov,cn=services,cn=acco
  unts,dc=itmodev,dc=gov
krbLastSuccessfulAuth: 20150921160115Z
ipaKrbPrincipalAlias: ldap/comipa02.itmodev....@itmodev.gov
userCertificate:: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXREDACTEDXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
krbExtraData:: AAgBAA==
krbLastPwdChange: 20111020180803Z
krbPasswordExpiration: 19700101000000Z
krbPrincipalKey:: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXREDACTEDXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
krbLoginFailedCount: 0
objectClass: ipaobject
objectClass: top
objectClass: ipaservice
objectClass: pkiuser
objectClass: krbprincipal
objectClass: krbprincipalaux
objectClass: krbTicketPolicyAux
objectClass: ipakrbprincipal
managedBy: fqdn=comipa02.itmodev.gov,cn=computers,cn=accounts,dc=itmodev,dc=go
  v
krbPrincipalName: ldap/comipa02.itmodev....@itmodev.gov
ipaUniqueID: 739e600a-fb46-11e0-a4b9-5254004d91a8

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root@comipa02 ~]#

Ok. This sure looks like a SASL mapping issue - the map should be using krbPrincipalName instead of uid.



-----Original Message-----
From: Martin Babinsky [mailto:mbabi...@redhat.com]
Sent: Tuesday, November 10, 2015 11:39 AM
To: Gronde, Christopher (Contractor) <christopher.gro...@fincen.gov>; Rich Megginson 
<rmegg...@redhat.com>; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos authentication 
error)

On 11/10/2015 05:16 PM, Gronde, Christopher (Contractor) wrote:
Neither came back with anything

# ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b 
"dc=itmodev,dc=gov" '(uid=ldap/comipa01.itmodev.gov)'
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=itmodev,dc=gov> with scope subtree # filter:
(uid=ldap/comipa01.itmodev.gov) # requesting: ALL #

# search result
search: 2
result: 0 Success

# numResponses: 1
[root@comipa02 ~]# ldapsearch -x -h 172.16.100.161 -D "cn=directory
manager" -W -b "dc=itmodev,dc=gov" '(uid=ldap/*.gov)' uid Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=itmodev,dc=gov> with scope subtree # filter:
(uid=ldap/*.gov) # requesting: uid #

# search result
search: 2
result: 0 Success

# numResponses: 1

I think that you should search for 'krbprincipalname' attribute when you want 
to find out whether you have the proper Kerberos principal present, like so:

ldapsearch -Y GSSAPI -b 'dc=ipa,dc=test'
'(krbprincipalname=ldap/master1.ipa.test*)'

-----Original Message-----
From: freeipa-users-boun...@redhat.com
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Rich Megginson
Sent: Tuesday, November 10, 2015 11:04 AM
To: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)

On 11/10/2015 08:18 AM, Gronde, Christopher (Contractor) wrote:
Thank you!  I should have caught that...

I changed the log level and then restarted dirsrv and attempted to start 
krb5kdc and got the following...
<snip>

[10/Nov/2015:10:12:02 -0500] conn=5 fd=64 slot=64 connection from
172.16.100.208 to 172.16.100.161
[10/Nov/2015:10:12:02 -0500] conn=5 op=0 BIND dn="" method=sasl
version=3 mech=GSSAPI
[10/Nov/2015:10:12:03 -0500] conn=5 op=0 RESULT err=14 tag=97
nentries=0 etime=1, SASL bind in progress
[10/Nov/2015:10:12:03 -0500] conn=5 op=1 BIND dn="" method=sasl
version=3 mech=GSSAPI
[10/Nov/2015:10:12:03 -0500] conn=5 op=1 RESULT err=14 tag=97
nentries=0 etime=0, SASL bind in progress
[10/Nov/2015:10:12:03 -0500] conn=5 op=2 BIND dn="" method=sasl
version=3 mech=GSSAPI
[10/Nov/2015:10:12:03 -0500] conn=Internal op=-1 SRCH
base="dc=itmodev,dc=gov" scope=2
filter="(uid=ldap/comipa01.itmodev.gov)" attrs=ALL
[10/Nov/2015:10:12:03 -0500] conn=Internal op=-1 RESULT err=0 tag=48
nentries=0 etime=0
[10/Nov/2015:10:12:03 -0500] conn=5 op=2 RESULT err=49 tag=97
nentries=0
etime=0
[10/Nov/2015:10:12:03 -0500] conn=5 op=3 UNBIND
[10/Nov/2015:10:12:03 -0500] conn=5 op=3 fd=64 closed - U1

<snip>

This is the SASL bind.  It thinks the principal in the Kerberos
credential is "ldap/comipa01.itmodev.gov", and the SASL map tells the
code to look for something with uid=ldap/comipa01.itmodev.gov under
dc=itmodev,dc=gov.  However, this entry is not found: RESULT err=0
tag=48 nentries=0.  nentries=0 means no entries matched the search criteria.

You can do the search yourself with ldapsearch:

ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b 
"dc=itmodev,dc=gov" '(uid=ldap/comipa01.itmodev.gov)'

If you want to find out if there is some other ldap principal, do a search like 
this:

ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b
"dc=itmodev,dc=gov" '(uid=ldap/*.gov)' uid

Ran into an error trying to set that

# ldapmodify -a -D "cn=directory manager" -W Enter LDAP Password:
dn: cn=config
changetype: modify
replace: nsslapd-acesslog-level
: 260

modifying entry "cn=config"
ldap_modify: Server is unwilling to perform (53)
            additional info: Unknown attribute nsslapd-acesslog-level
will be ignored

[root@comipa02 ~]# ldapmodify -a -D "cn=config" -W Enter LDAP
Password:
ldap_bind: Inappropriate authentication (48)

-----Original Message-----
From: Ludwig Krispenz [mailto:lkris...@redhat.com]
Sent: Tuesday, November 10, 2015 9:48 AM
To: Gronde, Christopher (Contractor) <christopher.gro...@fincen.gov>
Cc: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)


On 11/10/2015 03:32 PM, Gronde, Christopher (Contractor) wrote:
How do I change that log setting?  Is that done in LDAP?  Using ldapmodify?
yes,
ldapmodify ...
dn: cn=config
changetype: modify
replace: nsslapd-acesslog-level
nsslapd-acesslog-level: 260
-----Original Message-----
From: freeipa-users-boun...@redhat.com
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Ludwig
Krispenz
Sent: Tuesday, November 10, 2015 9:03 AM
To: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)


On 11/10/2015 02:40 PM, Alexander Bokovoy wrote:
On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:
Where can I verify or change the credentials it is trying to use?
Is it my LDAP password?
No, according to your logs, it is your LDAP master trying to
replicate (push changes) to your LDAP replica:
[09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection
from <MASTER_IP> to <REPLICA_IP>
[09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
version=3 mech=GSSAPI
err=49 could also be a result if the entry which is mapped from the principal 
is not found in the directory. A bit more info could be gained by enabling 
logging of internal searches.
Set nsslapd-acesslog-level: 260

and then look what internal searches are done during the gssapi
authentication
If that is true, it would be ldap/<master> Kerberos principal
talking to ldap/<replica> Kerberos principal. If that fails, it
means master and replica KDCs have different understanding of both
ldap/<master> and ldap/<replica> keys which most likely means keys
were rotated on master and weren't propagated to replica.

How to solve it? One possibility is to set master's hostname as
KDC address in krb5.conf on replica, forcing LDAP server on
replica to use master's KDC. I'm absolutely not sure this will
actually work but at least it allows to see if we are indeed
dealing with inconsistent state of service principals' keys.

-----Original Message-----
From: Alexander Bokovoy [mailto:aboko...@redhat.com]
Sent: Tuesday, November 10, 2015 8:18 AM
To: Gronde, Christopher (Contractor)
<christopher.gro...@fincen.gov>
Cc: Rob Crittenden <rcrit...@redhat.com>;
freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)

On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:
When I tried to start the service again I got no response from
tail of the log, but this is a repeating entry I see in the
access log

[09/Nov/2015:15:01:04 -0500] conn=1 fd=64 slot=64 connection
from
127.0.0.1 to 127.0.0.1
[09/Nov/2015:15:01:04 -0500] conn=1 op=-1 fd=64 closed - B1
[09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection
from <MASTER_IP> to <REPLICA_IP>
[09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
version=3 mech=GSSAPI
[09/Nov/2015:15:02:01 -0500] conn=2 op=0 RESULT err=14 tag=97
nentries=0 etime=0, SASL bind in progress
[09/Nov/2015:15:02:01 -0500] conn=2 op=1 BIND dn="" method=sasl
version=3 mech=GSSAPI
[09/Nov/2015:15:02:01 -0500] conn=2 op=1 RESULT err=14 tag=97
nentries=0 etime=0, SASL bind in progress
[09/Nov/2015:15:02:01 -0500] conn=2 op=2 BIND dn="" method=sasl
version=3 mech=GSSAPI
[09/Nov/2015:15:02:01 -0500] conn=2 op=2 RESULT err=49 tag=97
nentries=0 etime=0
[09/Nov/2015:15:02:01 -0500] conn=2 op=3 UNBIND
[09/Nov/2015:15:02:01 -0500] conn=2 op=3 fd=64 closed - U1

Does anyone know what err=14 or err=49 are?
err=14 means SASL bind in progress -- i.e. multi-round processing
is ongoing. This is normal for SASL GSSAPI.

err=49 is wrong password or username, i.e. credentials were incorrect.
It may also mean that LDAP server side was unable to process
Kerberos negotiation due to not having a current Kerberos ticket
for own service
(LDAP) and trying to request it from the Kerberos KDC but
Kerberos KDC is down.

-----Original Message-----
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Monday, November 09, 2015 3:26 PM
To: Gronde, Christopher (Contractor)
<christopher.gro...@fincen.gov>; Alexander Bokovoy
<aboko...@redhat.com>
Cc: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)

Gronde, Christopher (Contractor) wrote:
Nothing bad came back and there is definitely data in the tree.
Ok, I guess I'd try to start the kdc again and then watch the
389-ds access log (buffered) to:

1. See if it is binding at all
2. See what the search is and what, if any, results were
returned

This would be in /var/log/dirsrv/slapd-YOUR_REALM/access

rob

-----Original Message-----
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Monday, November 09, 2015 11:46 AM
To: Gronde, Christopher (Contractor)
<christopher.gro...@fincen.gov>; Alexander Bokovoy
<aboko...@redhat.com>
Cc: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)

Gronde, Christopher (Contractor) wrote:
I restarted dirsrv and attempted to start krb5kdc and this is
what the error log shows

# tail /var/log/dirsrv/slapd-ITMODEV-GOV/errors
[09/Nov/2015:11:01:02 -0500] - WARNING: userRoot: entry cache
size 10485760B is less than db size 28016640B; We recommend to
increase the entry cache size nsslapd-cachememsize.
[09/Nov/2015:11:01:02 -0500] - slapd started.  Listening on
All Interfaces port 389 for LDAP requests
[09/Nov/2015:11:06:04 -0500] - slapd shutting down - signaling
operation threads
[09/Nov/2015:11:06:04 -0500] - slapd shutting down - closing
down internal subsystems and plugins
[09/Nov/2015:11:06:04 -0500] - Waiting for 4 database threads
to stop
[09/Nov/2015:11:06:04 -0500] - All database threads now
stopped
[09/Nov/2015:11:06:04 -0500] - slapd stopped.
[09/Nov/2015:11:14:20 -0500] - 389-Directory/1.2.11.15
B2015.247.1737 starting up
[09/Nov/2015:11:14:20 -0500] - WARNING: userRoot: entry cache
size 10485760B is less than db size 28016640B; We recommend to
increase the entry cache size nsslapd-cachememsize.
[09/Nov/2015:11:14:20 -0500] - slapd started.  Listening on
All Interfaces port 389 for LDAP requests
Ok, that's good.

I'd do something like this to see what is in the db (substitute
example.com with your domain):

$ ldapsearch -x -D 'cn=Directory Manager' -W -s one -b
cn=kerberos,dc=example,dc=com

(don't post the output as it would include the kerberos master key).

If that returns nothing that's bad.

If it succeeds I'd broaden the search base a bit to see what
data you do
have:

$ ldapsearch -x -D 'cn=Directory Manager' -W -b
cn=groups,cn=accounts,dc=example,dc=com

I picked groups because usually groups << users in numbers.
This is just to see if you have data in the tree.

Let us know if either or both turns up nothing.

rob

-----Original Message-----
From: Alexander Bokovoy [mailto:aboko...@redhat.com]
Sent: Monday, November 09, 2015 10:51 AM
To: Gronde, Christopher (Contractor)
<christopher.gro...@fincen.gov>
Cc: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)

On Mon, 09 Nov 2015, Gronde, Christopher (Contractor) wrote:
Hello all!

On my replica IPA server after fixing a cert issue that had
been going on for sometime, I have all my certs figured out
but the krb5kdc service will not start.

# service krb5kdc start
Starting Kerberos 5 KDC: krb5kdc: cannot initialize realm
ITMODEV.GOV - see log file for details                  [FAILED]

# cat /var/log/krb5kdc.log
krb5kdc: Server error - while fetching master key K/M for
realm ITMODEV.GOV
krb5kdc: Server error - while fetching master key K/M for
realm ITMODEV.GOV
krb5kdc: Server error - while fetching master key K/M for
realm ITMODEV.GOV

I found this article online:
http://research.imb.uq.edu.au/~l.rathbone/ldap/kerberos.shtml

Which stated it might be because The slave KDC does not have
a stash file (.k5.EXAMPLE.COM). You need to create one. Tried
the command
listed:

# kdb5_util stash
kdb5_util: Server error while retrieving master entry

No further information found on the proceeding error above
for the kdb5_util command.

Any thoughts?
First: don't use instructions which are not related to IPA, please.

FreeIPA has its own LDAP driver for KDC and instructions for
anything else do not apply here at all.

If you see 'Server error - while fetching master key ..' it
means KDC LDAP driver was unable to contact LDAP server. Does
LDAP server work on the replica? What is in its error log
(/var/log/dirsrv/slapd-ITMODEV-GOV/errors)?

--
/ Alexander Bokovoy


--
/ Alexander Bokovoy

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project



--
Martin^3 Babinsky


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to