Hello Everyone
I recently lost the master master IPA server setup by the previous
administrator.
As it stands now, if I try to add a new client, in order to standup a
new replica, I get errors while trying to setup DNS. This led me to
look at how authentication worked (I'm new to IPA) and I learned about
the kerberos tools
I don't know if I'm familiar enough with the terminology to adequately
describe what I'm experiencing, so I'll give you some of the commands
and their results
but first, a bit on the design
before I got to this, we had
a <-> b <-> c <-> d
b was the master master
a, happened to point to two test servers nyc02ipa01 and nyc02ipa02
(not pictured, I discovered them later when c and d started having
problems)
a - nyc01ipa02
b - nyc01ipa01
c - ld4ipa01
d - ld4ipa02
currently, I have nyc02ipa02 <-> nyc01ipa02
the reason I have it limited like this is because all the other
servers stopped replicating for one reason or another (mainly that
they can't authenticate or in one case, there was a database record
corruption)
Anyway, here are some activities and logs from the latest round of
fixes and information activities I've been engaging in
22:54:32 root@nyc01ipa02:~# kinit admin
kinit: Clients credentials have been revoked while getting initial
credentials
Reading through this
<http://web.mit.edu/Kerberos/krb5-1.13/doc/admin/lockout.html> tells
me that
# kadmin: modprinc -unlock PRINCNAME
will unlock an account...but if I can't get in....
22:54:37 root@nyc01ipa02:~# kadmin
Authenticating as principal root/admin@MF with password.
kadmin: Client 'root/admin@MF' not found in Kerberos database
while initializing kadmin interface
on ld4ipa02, did a
# ipa-client-install --uninstall
then
# ipa-client-install --force-join --enable-dns-updates --permit -f
--ssh-trust-dns --request-cert --automount-location=LD4
--enable-dns-updates
DNS did not update, here is the relevant portion from
/var/log/ipaclient-install.log
2017-02-20T18:46:49Z DEBUG Writing nsupdate commands to
/etc/ipa/.dns_update.txt:
2017-02-20T18:46:49Z DEBUG debug
update delete ld4ipa02.mf. IN A
show
send
update delete ld4ipa02.mf. IN AAAA
show
send
update add ld4ipa02.mf. 1200 IN A 10.102.100.140
show
send
2017-02-20T18:46:49Z DEBUG Starting external process
2017-02-20T18:46:49Z DEBUG args=/usr/bin/nsupdate -g
/etc/ipa/.dns_update.txt
2017-02-20T18:46:49Z DEBUG Process finished, return code=1
2017-02-20T18:46:49Z DEBUG stdout=Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; UPDATE SECTION:
ld4ipa02.mf. 0 ANY A
2017-02-20T18:46:49Z DEBUG stderr=Reply from SOA query:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34702
;; flags: qr aa rd ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;ld4ipa02.mf. IN SOA
;; AUTHORITY SECTION:
mf. 1800 IN SOA ld4ipa01.mf. hostmaster.mf. 1487615509 3600 900 1209600 3600
Found zone name: mf
The master is: ld4ipa01.mf
start_gssrequest
tkey query failed: GSSAPI error: Major = Unspecified GSS failure. Minor
code may provide more information, Minor = Server DNS/ld4ipa01.mf@MF not found
in Kerberos database.
2017-02-20T18:46:49Z DEBUG nsupdate failed: Command '/usr/bin/nsupdate -g
/etc/ipa/.dns_update.txt' returned non-zero exit status 1
2017-02-20T18:46:49Z ERROR Failed to update DNS records.
2017-02-20T18:46:49Z DEBUG DNS resolver: Query: ld4ipa02.mf IN A
2017-02-20T18:46:49Z DEBUG DNS resolver: No record.
2017-02-20T18:46:49Z DEBUG DNS resolver: Query: ld4ipa02.mf IN AAAA
2017-02-20T18:46:49Z DEBUG DNS resolver: No record.
2017-02-20T18:46:49Z DEBUG DNS resolver: Query:140.100.102.10.in-addr.arpa
<http://140.100.102.10.in-addr.arpa/>. IN PTR
2017-02-20T18:46:49Z DEBUG DNS resolver: No record.
2017-02-20T18:46:49Z WARNING Missing A/AAAA record(s) for host ld4ipa02.mf:
10.102.100.140.
2017-02-20T18:46:49Z WARNING Missing reverse record(s) for address(es):
10.102.100.140.
Why isn't there an entry for "DNS/ld4ipa01.mf@MF" in the Kerberos
database?
klist -ktK /etc/dirsrv/ds.keytab on ld4ipa01 returns
Keytab name: FILE:/etc/dirsrv/ds.keytab
<http://file/etc/dirsrv/ds.keytab>
KVNO Timestamp Principal
---- -------------------
------------------------------------------------------
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(0x696a502bc73d209acdd36c42242f7f8aff9dbba1073b34ea018ed3bd9cdfd970)
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(0xe031464b6948ea34f4291d40fca7a21e)
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(0xe94a1c98fe79b6317901435d9e9e0257cefe438ff2ec527f)
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(0x6aaf4c7fa6b51b9de032b7c6428307b5)
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(0x5e0702f44aef9e0633e09eede7ca8041)
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(0x6e3a9d29ee3f129a156ae6228ab7728df8ce5de923a61eba6a2e7802b8d230b6)
Tried to test connectivity using ldapsearch found that I could
connect to other hosts on 389 but not 636
# ldapsearch -Hldap://nyc02ipa02:389 -D "cn=directory manager" -W -b "" -s
base
# ldapsearch -Hldaps://nyc02ipa02:686 -D "cn=directory manager" -W -b "" -s
base
Testing the kvno
02:10:00 root@ld4ipa01:~# kvno DNS/ld4ipa01.mf@MF
DNS/ld4ipa01.mf@MF: kvno = 2
02:10:52 root@ld4ipa02:~# kvno DNS/ld4ipa01.mf@MF
kvno: Server DNS/ld4ipa01.mf@MF not found in Kerberos database
while getting credentials for DNS/ld4ipa01.mf@MF
Add this to any command line to get debug on kerberos commands
KRB5_TRACE=/dev/stdout kvno DNS/ld4ipa01.mf@MF
So, looking at the debug
kvno from ld4ipa02, does not return tickets. It does this because it
contacts the KDC which is nyc02ipa02, and nyc02ipa02 does not
recognize ldipa02 as an IPA server. It doesn't recognize ld4ipa01 either.
right now, if I try to connect nyc02ipa02 to ld4ipa01 I get
21:56:27 root@nyc02ipa02:~# ipa topologysegment-add domain
ld4ipa01-to-nyc02ipa02 --leftnode ld4ipa01.mf --rightnode
nyc02ipa02.mf
ipa: ERROR: invalid 'leftnode': left node is not a topology node:
ld4ipa01.mf
ipa privilege-show 'DNS Servers' --all --raw
dn: cn=DNS Servers,cn=privileges,cn=pbac,dc=mf
cn: DNS Servers
description: DNS Servers
member: krbprincipalname=DNS/nyc01ipa02.mf@MF,cn=services,cn=accounts,dc=mf
member:
krbprincipalname=ipa-dnskeysyncd/nyc01ipa02.mf@MF,cn=services,cn=accounts,dc=mf
member: krbprincipalname=DNS/nyc02ipa02.mf@MF,cn=services,cn=accounts,dc=mf
member:
krbprincipalname=ipa-dnskeysyncd/nyc02ipa02.mf@MF,cn=services,cn=accounts,dc=mf
member:
krbprincipalname=ipa-ods-exporter/nyc01ipa02.mf@MF,cn=services,cn=accounts,dc=mf
memberof: cn=System: Read DNS Configuration,cn=permissions,cn=pbac,dc=mf
memberof: cn=System: Write DNS Configuration,cn=permissions,cn=pbac,dc=mf
memberof: cn=System: Add DNS Entries,cn=permissions,cn=pbac,dc=mf
memberof: cn=System: Manage DNSSEC keys,cn=permissions,cn=pbac,dc=mf
memberof: cn=System: Manage DNSSEC metadata,cn=permissions,cn=pbac,dc=mf
memberof: cn=System: Read DNS Entries,cn=permissions,cn=pbac,dc=mf
memberof: cn=System: Remove DNS Entries,cn=permissions,cn=pbac,dc=mf
memberof: cn=System: Update DNS Entries,cn=permissions,cn=pbac,dc=mf
memberof: cn=System: Read DNS Servers
Configuration,cn=permissions,cn=pbac,dc=mf
objectClass: top
objectClass: groupofnames
objectClass: nestedgroup
--
Aaron Young
MarketFactory, Manager of Site Reliability Engineering
425 Broadway, 3FL
New York, NY 10013
Office: +1 212 625 9988
Direct +1 646 779 3710
US Support: +1 (212) 625-0688 <tel:%2B1%20%28212%29%20625-0688> | UK
Support: +44 (0) 203 695-7997 <tel:%2B44%20%280%29%20203%20695-7997>
