Re: [Freeipa-users] FreeIPA Replica / HA Issues

2016-01-14 Thread Jeff Hallyburton
Petr,

Thanks for the info.  This is in fact probably what's happening in our
case.  That said, is there any supported way of manually setting up
failover at this time?  Is it hard, or simply impossible?

Thanks,

Jeff

Jeff Hallyburton
Strategic Systems Engineer
Bloomip Inc.
Web: http://www.bloomip.com

Engineering Support: supp...@bloomip.com
Billing Support: bill...@bloomip.com
Customer Support Portal:  https://my.bloomip.com 

On Thu, Jan 14, 2016 at 2:06 AM, Petr Spacek  wrote:

> Hello,
>
>
> this log is weird:
>
> On 14.1.2016 03:02, Jeff Hallyburton wrote:
> >> 2016-01-14T00:45:35Z DEBUG [IPA Discovery]
> >> 2016-01-14T00:45:35Z DEBUG Starting IPA discovery with domain=
> west-2.production.example.com, servers=None, hostname=
> test.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG Search for LDAP SRV record in
> west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG Search DNS for SRV record of _ldap._
> tcp.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: 0 100 389
> ipa1.west-2.production.example.com.
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: 10 100 389
> ipa2.west-2.production.example.com.
> >> 2016-01-14T00:45:35Z DEBUG [Kerberos realm search]
> >> 2016-01-14T00:45:35Z DEBUG Search DNS for TXT record of _
> kerberos.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: "EXAMPLE.COM"
> >> 2016-01-14T00:45:35Z DEBUG Search DNS for SRV record of _kerberos._
> udp.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: 10 100 88
> ipa2.west-2.production.example.com.
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: 0 100 88
> ipa1.west-2.production.example.com.
> >> 2016-01-14T00:45:35Z DEBUG [LDAP server check]
> >> 2016-01-14T00:45:35Z DEBUG Verifying that
> ipa1.west-2.production.example.com (realm EXAMPLE.COM) is an IPA server
> >> 2016-01-14T00:45:35Z DEBUG Init LDAP connection to:
> ipa1.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG Search LDAP server for IPA base DN
> >> 2016-01-14T00:45:35Z DEBUG Check if naming context 'dc=example,dc=com'
> is for IPA
> >> 2016-01-14T00:45:35Z DEBUG Naming context 'dc=example,dc=com' is a
> valid IPA context
> >> 2016-01-14T00:45:35Z DEBUG Search for (objectClass=krbRealmContainer)
> in dc=example,dc=com (sub)
> >> 2016-01-14T00:45:35Z DEBUG Found: cn=EXAMPLE.COM
> ,cn=kerberos,dc=example,dc=com
> >> 2016-01-14T00:45:35Z DEBUG Discovery result: Success; server=
> ipa1.west-2.production.example.com, domain=west-2.production.example.com,
> kdc=ipa2.west-2.production.example.com,ipa1.west-2.production.example.com,
> basedn=dc=example,dc=com
> >> 2016-01-14T00:45:35Z DEBUG Validated servers:
> ipa1.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG will use discovered domain:
> west-2.production.example.com
>
> It looks that your IPA domain & realm is "example.com" and "EXAMPLE.COM",
> is
> that correct?
>
> Looking further ...
>
> > 2016-01-14T00:45:39Z DEBUG Writing Kerberos configuration to
> /etc/krb5.conf:
> > 2016-01-14T00:45:39Z DEBUG #File modified by ipa-client-install
> >
> > includedir /var/lib/sss/pubconf/krb5.include.d/
> >
> > [libdefaults]
> >   default_realm = EXAMPLE.COM
> >   dns_lookup_realm = true
> >   dns_lookup_kdc = true
> >   rdns = false
> >   ticket_lifetime = 24h
> >   forwardable = yes
> >   udp_preference_limit = 0
> >   default_ccache_name = KEYRING:persistent:%{uid}
> >
> >
> > [realms]
> >   EXAMPLE.COM = {
> > pkinit_anchors = FILE:/etc/ipa/ca.crt
> >
> >   }
> >
> >
> > [domain_realm]
> >   .west-2.production.example.com = EXAMPLE.COM
> >   west-2.production.example.com = EXAMPLE.COM
>
> Hmm, this is going to be wild guess, but let's try it:
> Do you have DNS SRV records in domain west-2.production.example.com but
> not in
> DNS domain example.com?
>
> That would probably cause this kind of problem.
>
> Generally it is necessary to put _kerberos TXT + SRV records into the
> (primary) DNS domain specified during IPA installation. Then use --domain
> option during ipa-client-install.
>
> --server is generally discouraged as it disables DNS SRV lookup and makes
> failover hard or impossible.
>
> --domain is just a hint for the installer where to start looking for DNS
> SRV
> records and allows full automatic failover.
>
>
> The autodiscovery is quite messy and needs to be imporoved in next
> versions.
> https://fedorahosted.org/freeipa/ticket/5270 should avoid the need to
> specify
> --domain when Kerberos TXT record is in DNS ... Stay tuned :-)
>
> --
> Petr^2 Spacek
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] FreeIPA Replica / HA Issues

2016-01-14 Thread Petr Spacek
Hello,

On 15.1.2016 02:59, Jeff Hallyburton wrote:
> Petr,
> 
> Thanks for the info.  This is in fact probably what's happening in our
> case.  That said, is there any supported way of manually setting up
> failover at this time?  Is it hard, or simply impossible?

The supported (and cleanest) way is to add SRV records to the domain equal to
Kerberos realm. Technically nothing prevents you from doing so even 
post-install.

All other configurations are non-standard, depend heavily on client, and may
blow up in some situations. If you are using SSSD, try to set
dns_discovery_domain option in sssd.conf to the domain name which holds all
SRV records. It should help, but again, all other clients may blow up
occasionally.

Petr Spacek @ Red Hat

> On Thu, Jan 14, 2016 at 2:06 AM, Petr Spacek  wrote:
> 
>> Hello,
>>
>>
>> this log is weird:
>>
>> On 14.1.2016 03:02, Jeff Hallyburton wrote:
 2016-01-14T00:45:35Z DEBUG [IPA Discovery]
 2016-01-14T00:45:35Z DEBUG Starting IPA discovery with domain=
>> west-2.production.example.com, servers=None, hostname=
>> test.west-2.production.example.com
 2016-01-14T00:45:35Z DEBUG Search for LDAP SRV record in
>> west-2.production.example.com
 2016-01-14T00:45:35Z DEBUG Search DNS for SRV record of _ldap._
>> tcp.west-2.production.example.com
 2016-01-14T00:45:35Z DEBUG DNS record found: 0 100 389
>> ipa1.west-2.production.example.com.
 2016-01-14T00:45:35Z DEBUG DNS record found: 10 100 389
>> ipa2.west-2.production.example.com.
 2016-01-14T00:45:35Z DEBUG [Kerberos realm search]
 2016-01-14T00:45:35Z DEBUG Search DNS for TXT record of _
>> kerberos.west-2.production.example.com
 2016-01-14T00:45:35Z DEBUG DNS record found: "EXAMPLE.COM"
 2016-01-14T00:45:35Z DEBUG Search DNS for SRV record of _kerberos._
>> udp.west-2.production.example.com
 2016-01-14T00:45:35Z DEBUG DNS record found: 10 100 88
>> ipa2.west-2.production.example.com.
 2016-01-14T00:45:35Z DEBUG DNS record found: 0 100 88
>> ipa1.west-2.production.example.com.
 2016-01-14T00:45:35Z DEBUG [LDAP server check]
 2016-01-14T00:45:35Z DEBUG Verifying that
>> ipa1.west-2.production.example.com (realm EXAMPLE.COM) is an IPA server
 2016-01-14T00:45:35Z DEBUG Init LDAP connection to:
>> ipa1.west-2.production.example.com
 2016-01-14T00:45:35Z DEBUG Search LDAP server for IPA base DN
 2016-01-14T00:45:35Z DEBUG Check if naming context 'dc=example,dc=com'
>> is for IPA
 2016-01-14T00:45:35Z DEBUG Naming context 'dc=example,dc=com' is a
>> valid IPA context
 2016-01-14T00:45:35Z DEBUG Search for (objectClass=krbRealmContainer)
>> in dc=example,dc=com (sub)
 2016-01-14T00:45:35Z DEBUG Found: cn=EXAMPLE.COM
>> ,cn=kerberos,dc=example,dc=com
 2016-01-14T00:45:35Z DEBUG Discovery result: Success; server=
>> ipa1.west-2.production.example.com, domain=west-2.production.example.com,
>> kdc=ipa2.west-2.production.example.com,ipa1.west-2.production.example.com,
>> basedn=dc=example,dc=com
 2016-01-14T00:45:35Z DEBUG Validated servers:
>> ipa1.west-2.production.example.com
 2016-01-14T00:45:35Z DEBUG will use discovered domain:
>> west-2.production.example.com
>>
>> It looks that your IPA domain & realm is "example.com" and "EXAMPLE.COM",
>> is
>> that correct?
>>
>> Looking further ...
>>
>>> 2016-01-14T00:45:39Z DEBUG Writing Kerberos configuration to
>> /etc/krb5.conf:
>>> 2016-01-14T00:45:39Z DEBUG #File modified by ipa-client-install
>>>
>>> includedir /var/lib/sss/pubconf/krb5.include.d/
>>>
>>> [libdefaults]
>>>   default_realm = EXAMPLE.COM
>>>   dns_lookup_realm = true
>>>   dns_lookup_kdc = true
>>>   rdns = false
>>>   ticket_lifetime = 24h
>>>   forwardable = yes
>>>   udp_preference_limit = 0
>>>   default_ccache_name = KEYRING:persistent:%{uid}
>>>
>>>
>>> [realms]
>>>   EXAMPLE.COM = {
>>> pkinit_anchors = FILE:/etc/ipa/ca.crt
>>>
>>>   }
>>>
>>>
>>> [domain_realm]
>>>   .west-2.production.example.com = EXAMPLE.COM
>>>   west-2.production.example.com = EXAMPLE.COM
>>
>> Hmm, this is going to be wild guess, but let's try it:
>> Do you have DNS SRV records in domain west-2.production.example.com but
>> not in
>> DNS domain example.com?
>>
>> That would probably cause this kind of problem.
>>
>> Generally it is necessary to put _kerberos TXT + SRV records into the
>> (primary) DNS domain specified during IPA installation. Then use --domain
>> option during ipa-client-install.
>>
>> --server is generally discouraged as it disables DNS SRV lookup and makes
>> failover hard or impossible.
>>
>> --domain is just a hint for the installer where to start looking for DNS
>> SRV
>> records and allows full automatic failover.
>>
>>
>> The autodiscovery is quite messy and needs to be imporoved in next
>> versions.
>> https://fedorahosted.org/freeipa/ticket/5270 should avoid the need to
>> specify
>> --domain when Kerberos TXT record is in DNS ... Stay tuned :-)
>>
>> --
>> Petr^2 Spacek

-- 
Manage your 

Re: [Freeipa-users] FreeIPA Replica / HA Issues

2016-01-13 Thread Petr Spacek
Hello,


this log is weird:

On 14.1.2016 03:02, Jeff Hallyburton wrote:
>> 2016-01-14T00:45:35Z DEBUG [IPA Discovery]
>> 2016-01-14T00:45:35Z DEBUG Starting IPA discovery with 
>> domain=west-2.production.example.com, servers=None, 
>> hostname=test.west-2.production.example.com
>> 2016-01-14T00:45:35Z DEBUG Search for LDAP SRV record in 
>> west-2.production.example.com
>> 2016-01-14T00:45:35Z DEBUG Search DNS for SRV record of 
>> _ldap._tcp.west-2.production.example.com
>> 2016-01-14T00:45:35Z DEBUG DNS record found: 0 100 389 
>> ipa1.west-2.production.example.com.
>> 2016-01-14T00:45:35Z DEBUG DNS record found: 10 100 389 
>> ipa2.west-2.production.example.com.
>> 2016-01-14T00:45:35Z DEBUG [Kerberos realm search]
>> 2016-01-14T00:45:35Z DEBUG Search DNS for TXT record of 
>> _kerberos.west-2.production.example.com
>> 2016-01-14T00:45:35Z DEBUG DNS record found: "EXAMPLE.COM"
>> 2016-01-14T00:45:35Z DEBUG Search DNS for SRV record of 
>> _kerberos._udp.west-2.production.example.com
>> 2016-01-14T00:45:35Z DEBUG DNS record found: 10 100 88 
>> ipa2.west-2.production.example.com.
>> 2016-01-14T00:45:35Z DEBUG DNS record found: 0 100 88 
>> ipa1.west-2.production.example.com.
>> 2016-01-14T00:45:35Z DEBUG [LDAP server check]
>> 2016-01-14T00:45:35Z DEBUG Verifying that ipa1.west-2.production.example.com 
>> (realm EXAMPLE.COM) is an IPA server
>> 2016-01-14T00:45:35Z DEBUG Init LDAP connection to: 
>> ipa1.west-2.production.example.com
>> 2016-01-14T00:45:35Z DEBUG Search LDAP server for IPA base DN
>> 2016-01-14T00:45:35Z DEBUG Check if naming context 'dc=example,dc=com' is 
>> for IPA
>> 2016-01-14T00:45:35Z DEBUG Naming context 'dc=example,dc=com' is a valid IPA 
>> context
>> 2016-01-14T00:45:35Z DEBUG Search for (objectClass=krbRealmContainer) in 
>> dc=example,dc=com (sub)
>> 2016-01-14T00:45:35Z DEBUG Found: 
>> cn=EXAMPLE.COM,cn=kerberos,dc=example,dc=com
>> 2016-01-14T00:45:35Z DEBUG Discovery result: Success; 
>> server=ipa1.west-2.production.example.com, 
>> domain=west-2.production.example.com, 
>> kdc=ipa2.west-2.production.example.com,ipa1.west-2.production.example.com, 
>> basedn=dc=example,dc=com
>> 2016-01-14T00:45:35Z DEBUG Validated servers: 
>> ipa1.west-2.production.example.com
>> 2016-01-14T00:45:35Z DEBUG will use discovered domain: 
>> west-2.production.example.com

It looks that your IPA domain & realm is "example.com" and "EXAMPLE.COM", is
that correct?

Looking further ...

> 2016-01-14T00:45:39Z DEBUG Writing Kerberos configuration to /etc/krb5.conf:
> 2016-01-14T00:45:39Z DEBUG #File modified by ipa-client-install
> 
> includedir /var/lib/sss/pubconf/krb5.include.d/
> 
> [libdefaults]
>   default_realm = EXAMPLE.COM
>   dns_lookup_realm = true
>   dns_lookup_kdc = true
>   rdns = false
>   ticket_lifetime = 24h
>   forwardable = yes
>   udp_preference_limit = 0
>   default_ccache_name = KEYRING:persistent:%{uid}
> 
> 
> [realms]
>   EXAMPLE.COM = {
> pkinit_anchors = FILE:/etc/ipa/ca.crt
> 
>   }
> 
> 
> [domain_realm]
>   .west-2.production.example.com = EXAMPLE.COM
>   west-2.production.example.com = EXAMPLE.COM

Hmm, this is going to be wild guess, but let's try it:
Do you have DNS SRV records in domain west-2.production.example.com but not in
DNS domain example.com?

That would probably cause this kind of problem.

Generally it is necessary to put _kerberos TXT + SRV records into the
(primary) DNS domain specified during IPA installation. Then use --domain
option during ipa-client-install.

--server is generally discouraged as it disables DNS SRV lookup and makes
failover hard or impossible.

--domain is just a hint for the installer where to start looking for DNS SRV
records and allows full automatic failover.


The autodiscovery is quite messy and needs to be imporoved in next versions.
https://fedorahosted.org/freeipa/ticket/5270 should avoid the need to specify
--domain when Kerberos TXT record is in DNS ... Stay tuned :-)

-- 
Petr^2 Spacek

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


Re: [Freeipa-users] FreeIPA Replica / HA Issues

2016-01-13 Thread Rob Crittenden
Jeff Hallyburton wrote:
> We've deployed a FreeIPA server in a client infrastructure and now we're
> working on making that setup HA.  We've created a replica and I can
> verify that the replica has connectivity to the existing master and
> ensured that the auto-discovery DNS records are set up for LDAP /
> Kerberos / etc, but I'm having a couple of issues with clients:  
> 
> 1.  ipa-client-install fails with the following error whenever a server
> is not explicitly specified (though explicitly specifying either the
> original master OR the replica works fine):
> 
> trying https://ipa1.west-2.production.example.com/ipa/json
> 
> Cannot connect to the server due to Kerberos error: Kerberos error:
> Kerberos error: ('Unspecified GSS failure.  Minor code may provide more
> information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> "', -1765328230)/. Trying with delegate=True
> 
> trying https://ipa1.west-2.production.example.com/ipa/json
> 
> Second connect with delegate=True also failed: Kerberos error: Kerberos
> error: ('Unspecified GSS failure.  Minor code may provide more
> information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> "', -1765328230)/
> 
> Cannot connect to the IPA server RPC interface: Kerberos error: Kerberos
> error: ('Unspecified GSS failure.  Minor code may provide more
> information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> "', -1765328230)/
> 
> Installation failed. Rolling back changes.
> 
> Failed to list certificates in /etc/ipa/nssdb: Command
> ''/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-L'' returned non-zero exit
> status 255
> 
> Unenrolling client from IPA server
> 
> Unenrolling host failed: Error obtaining initial credentials: Cannot
> find KDC for requested realm.
> 
> 
> What we see in the install logs is:
> 
> 2016-01-14T00:45:39Z INFO Configured /etc/krb5.conf for IPA realm
> EXAMPLE.COM 
> 
> 2016-01-14T00:45:39Z DEBUG Starting external process
> 
> 2016-01-14T00:45:39Z DEBUG args='keyctl' 'search' '@s' 'user'
> 'ipa_session_cookie:host/test.west-2.production.example@example.com
> '
> 
> 2016-01-14T00:45:39Z DEBUG Process finished, return code=1
> 
> 2016-01-14T00:45:39Z DEBUG stdout=
> 
> 2016-01-14T00:45:39Z DEBUG stderr=keyctl_search: Required key not available
> 
> 
> 2016-01-14T00:45:39Z DEBUG Starting external process
> 
> 2016-01-14T00:45:39Z DEBUG args='/usr/bin/certutil' '-d'
> '/tmp/tmpCJNEzU' '-N' '-f' '/tmp/tmpPN7H8R'
> 
> 2016-01-14T00:45:39Z DEBUG Process finished, return code=0
> 
> 2016-01-14T00:45:39Z DEBUG stdout=
> 
> 2016-01-14T00:45:39Z DEBUG stderr=
> 
> 2016-01-14T00:45:39Z DEBUG Starting external process
> 
> 2016-01-14T00:45:39Z DEBUG args='/usr/bin/certutil' '-d'
> '/tmp/tmpCJNEzU' '-A' '-n' 'CA certificate 1' '-t' 'C,,'
> 
> 2016-01-14T00:45:39Z DEBUG Process finished, return code=0
> 
> 2016-01-14T00:45:39Z DEBUG stdout=
> 
> 2016-01-14T00:45:39Z DEBUG stderr=
> 
> 2016-01-14T00:45:39Z DEBUG Starting external process
> 
> 2016-01-14T00:45:39Z DEBUG args='keyctl' 'search' '@s' 'user'
> 'ipa_session_cookie:host/test.west-2.production.example@example.com
> '
> 
> 2016-01-14T00:45:39Z DEBUG Process finished, return code=1
> 
> 2016-01-14T00:45:39Z DEBUG stdout=
> 
> 2016-01-14T00:45:39Z DEBUG stderr=keyctl_search: Required key not available
> 
> 
> 2016-01-14T00:45:39Z DEBUG failed to find session_cookie in persistent
> storage for principal
> 'host/test.west-2.production.example@example.com
> '
> 
> 2016-01-14T00:45:39Z INFO trying
> https://ipa1.west-2.production.example.com/ipa/json
> 
> 2016-01-14T00:45:39Z INFO Cannot connect to the server due to Kerberos
> error: Kerberos error: Kerberos error: ('Unspecified GSS failure.  Minor
> code may provide more information', 851968)/('Cannot find KDC for realm
> "EXAMPLE.COM "', -1765328230)/. Trying with
> delegate=True
> 
> 2016-01-14T00:45:39Z INFO trying
> https://ipa1.west-2.production.example.com/ipa/json
> 
> 2016-01-14T00:45:39Z WARNING Second connect with delegate=True also
> failed: Kerberos error: Kerberos error: ('Unspecified GSS failure. 
> Minor code may provide more information', 851968)/('Cannot find KDC for
> realm "EXAMPLE.COM "', -1765328230)/
> 
> 2016-01-14T00:45:39Z ERROR Cannot connect to the IPA server RPC
> interface: Kerberos error: Kerberos error: ('Unspecified GSS failure. 
> Minor code may provide more information', 851968)/('Cannot find KDC for
> realm "EXAMPLE.COM "', -1765328230)/
> 
> 2016-01-14T00:45:39Z ERROR Installation failed. Rolling back changes.
> 
> 2016-01-14T00:45:39Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 
> 2016-01-14T00:45:39Z DEBUG Starting external process
> 
> 

Re: [Freeipa-users] FreeIPA Replica / HA Issues

2016-01-13 Thread Jeff Hallyburton
Rob,

Full log is attached.

Jeff

Jeff Hallyburton
Strategic Systems Engineer
Bloomip Inc.
Web: http://www.bloomip.com

Engineering Support: supp...@bloomip.com
Billing Support: bill...@bloomip.com
Customer Support Portal:  https://my.bloomip.com 

On Wed, Jan 13, 2016 at 8:35 PM, Rob Crittenden  wrote:

> Jeff Hallyburton wrote:
> > We've deployed a FreeIPA server in a client infrastructure and now we're
> > working on making that setup HA.  We've created a replica and I can
> > verify that the replica has connectivity to the existing master and
> > ensured that the auto-discovery DNS records are set up for LDAP /
> > Kerberos / etc, but I'm having a couple of issues with clients:
> >
> > 1.  ipa-client-install fails with the following error whenever a server
> > is not explicitly specified (though explicitly specifying either the
> > original master OR the replica works fine):
> >
> > trying https://ipa1.west-2.production.example.com/ipa/json
> >
> > Cannot connect to the server due to Kerberos error: Kerberos error:
> > Kerberos error: ('Unspecified GSS failure.  Minor code may provide more
> > information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> > "', -1765328230)/. Trying with delegate=True
> >
> > trying https://ipa1.west-2.production.example.com/ipa/json
> >
> > Second connect with delegate=True also failed: Kerberos error: Kerberos
> > error: ('Unspecified GSS failure.  Minor code may provide more
> > information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> > "', -1765328230)/
> >
> > Cannot connect to the IPA server RPC interface: Kerberos error: Kerberos
> > error: ('Unspecified GSS failure.  Minor code may provide more
> > information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> > "', -1765328230)/
> >
> > Installation failed. Rolling back changes.
> >
> > Failed to list certificates in /etc/ipa/nssdb: Command
> > ''/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-L'' returned non-zero exit
> > status 255
> >
> > Unenrolling client from IPA server
> >
> > Unenrolling host failed: Error obtaining initial credentials: Cannot
> > find KDC for requested realm.
> >
> >
> > What we see in the install logs is:
> >
> > 2016-01-14T00:45:39Z INFO Configured /etc/krb5.conf for IPA realm
> > EXAMPLE.COM 
> >
> > 2016-01-14T00:45:39Z DEBUG Starting external process
> >
> > 2016-01-14T00:45:39Z DEBUG args='keyctl' 'search' '@s' 'user'
> > 'ipa_session_cookie:host/test.west-2.production.example@example.com
> > '
> >
> > 2016-01-14T00:45:39Z DEBUG Process finished, return code=1
> >
> > 2016-01-14T00:45:39Z DEBUG stdout=
> >
> > 2016-01-14T00:45:39Z DEBUG stderr=keyctl_search: Required key not
> available
> >
> >
> > 2016-01-14T00:45:39Z DEBUG Starting external process
> >
> > 2016-01-14T00:45:39Z DEBUG args='/usr/bin/certutil' '-d'
> > '/tmp/tmpCJNEzU' '-N' '-f' '/tmp/tmpPN7H8R'
> >
> > 2016-01-14T00:45:39Z DEBUG Process finished, return code=0
> >
> > 2016-01-14T00:45:39Z DEBUG stdout=
> >
> > 2016-01-14T00:45:39Z DEBUG stderr=
> >
> > 2016-01-14T00:45:39Z DEBUG Starting external process
> >
> > 2016-01-14T00:45:39Z DEBUG args='/usr/bin/certutil' '-d'
> > '/tmp/tmpCJNEzU' '-A' '-n' 'CA certificate 1' '-t' 'C,,'
> >
> > 2016-01-14T00:45:39Z DEBUG Process finished, return code=0
> >
> > 2016-01-14T00:45:39Z DEBUG stdout=
> >
> > 2016-01-14T00:45:39Z DEBUG stderr=
> >
> > 2016-01-14T00:45:39Z DEBUG Starting external process
> >
> > 2016-01-14T00:45:39Z DEBUG args='keyctl' 'search' '@s' 'user'
> > 'ipa_session_cookie:host/test.west-2.production.example@example.com
> > '
> >
> > 2016-01-14T00:45:39Z DEBUG Process finished, return code=1
> >
> > 2016-01-14T00:45:39Z DEBUG stdout=
> >
> > 2016-01-14T00:45:39Z DEBUG stderr=keyctl_search: Required key not
> available
> >
> >
> > 2016-01-14T00:45:39Z DEBUG failed to find session_cookie in persistent
> > storage for principal
> > 'host/test.west-2.production.example@example.com
> > '
> >
> > 2016-01-14T00:45:39Z INFO trying
> > https://ipa1.west-2.production.example.com/ipa/json
> >
> > 2016-01-14T00:45:39Z INFO Cannot connect to the server due to Kerberos
> > error: Kerberos error: Kerberos error: ('Unspecified GSS failure.  Minor
> > code may provide more information', 851968)/('Cannot find KDC for realm
> > "EXAMPLE.COM "', -1765328230)/. Trying with
> > delegate=True
> >
> > 2016-01-14T00:45:39Z INFO trying
> > https://ipa1.west-2.production.example.com/ipa/json
> >
> > 2016-01-14T00:45:39Z WARNING Second connect with delegate=True also
> > failed: Kerberos error: Kerberos error: ('Unspecified GSS failure.
> > Minor code may provide more information', 851968)/('Cannot find KDC for
> > realm "EXAMPLE.COM "',