Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Martin Basti]

On 20.01.2016 15:45, Simo Sorce wrote:

On Wed, 2016-01-20 at 09:42 +0100, Martin Babinsky wrote:

On 01/15/2016 06:29 PM, Martin Babinsky wrote:

On 01/15/2016 04:57 PM, Simo Sorce wrote:

On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote:

On 01/14/2016 10:31 PM, Simo Sorce wrote:

On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote:

On 01/13/2016 10:31 AM, Martin Babinsky wrote:

On 01/07/2016 05:38 PM, Martin Babinsky wrote:

On 01/07/2016 05:37 PM, Martin Babinsky wrote:

https://fedorahosted.org/freeipa/ticket/5584


And the patch is here.




self-NACK, there may be a better way to handle this. I will do some
investigation and send updated patch.


Attaching updated patch.

A failure to obtain a tgt may be due to other reasons (for example the
KDC crashed), why are you trying to use this test ?
Isn't it sufficient to see there is no host entry in the directory ?

Simo.


There were some corner cases I encountered, mostly concerning a cleanup
after unsuccessful replica promotion.

You may sometimes end up in a state where local DS is working, but KDC
crashed and the krb5.conf is still pointing at a remote one. In that
case "malformed" replica's local host entry exist, but when such host
tries to get TGT, the AS-REQ goes to remote KDC from other master.

However, if the admin had in the mean time cleaned up this host's
kerberos principals/keys, the crashed replica gets one of the following
errors:

Client not found in Kerberos database
Client credentials have been revoked
Generic preauthentication failure

These were printed out as errors during uninstall, but were actually
expected in situation like this. It is true that the code should check
and ignore these specific errors.

Only the first id valid for your case, the others may be transient
errors.

Simo.



True, attaching updated patch. The other errors will now pop out in the
output and the warning will be displayed.




Bump for review.


LGTM
Simo.


ACK

Pushed to:
master: d726da3ba20283ffdc1d384dfedf8e6a732dc3d7
ipa-4-3: 4f0266f925207ca705b45287744b3e609d841cc6

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Martin Babinsky]

On 01/15/2016 06:29 PM, Martin Babinsky wrote:

On 01/15/2016 04:57 PM, Simo Sorce wrote:

On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote:

On 01/14/2016 10:31 PM, Simo Sorce wrote:

On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote:

On 01/13/2016 10:31 AM, Martin Babinsky wrote:

On 01/07/2016 05:38 PM, Martin Babinsky wrote:

On 01/07/2016 05:37 PM, Martin Babinsky wrote:

https://fedorahosted.org/freeipa/ticket/5584


And the patch is here.




self-NACK, there may be a better way to handle this. I will do some
investigation and send updated patch.


Attaching updated patch.


A failure to obtain a tgt may be due to other reasons (for example the
KDC crashed), why are you trying to use this test ?
Isn't it sufficient to see there is no host entry in the directory ?

Simo.


There were some corner cases I encountered, mostly concerning a cleanup
after unsuccessful replica promotion.

You may sometimes end up in a state where local DS is working, but KDC
crashed and the krb5.conf is still pointing at a remote one. In that
case "malformed" replica's local host entry exist, but when such host
tries to get TGT, the AS-REQ goes to remote KDC from other master.

However, if the admin had in the mean time cleaned up this host's
kerberos principals/keys, the crashed replica gets one of the following
errors:

Client not found in Kerberos database
Client credentials have been revoked
Generic preauthentication failure

These were printed out as errors during uninstall, but were actually
expected in situation like this. It is true that the code should check
and ignore these specific errors.


Only the first id valid for your case, the others may be transient
errors.

Simo.



True, attaching updated patch. The other errors will now pop out in the
output and the warning will be displayed.




Bump for review.

--
Martin^3 Babinsky

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Simo Sorce]

On Wed, 2016-01-20 at 09:42 +0100, Martin Babinsky wrote:
> On 01/15/2016 06:29 PM, Martin Babinsky wrote:
> > On 01/15/2016 04:57 PM, Simo Sorce wrote:
> >> On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote:
> >>> On 01/14/2016 10:31 PM, Simo Sorce wrote:
>  On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote:
> > On 01/13/2016 10:31 AM, Martin Babinsky wrote:
> >> On 01/07/2016 05:38 PM, Martin Babinsky wrote:
> >>> On 01/07/2016 05:37 PM, Martin Babinsky wrote:
>  https://fedorahosted.org/freeipa/ticket/5584
> 
> >>> And the patch is here.
> >>>
> >>>
> >>>
> >> self-NACK, there may be a better way to handle this. I will do some
> >> investigation and send updated patch.
> >>
> > Attaching updated patch.
> 
>  A failure to obtain a tgt may be due to other reasons (for example the
>  KDC crashed), why are you trying to use this test ?
>  Isn't it sufficient to see there is no host entry in the directory ?
> 
>  Simo.
> 
> >>> There were some corner cases I encountered, mostly concerning a cleanup
> >>> after unsuccessful replica promotion.
> >>>
> >>> You may sometimes end up in a state where local DS is working, but KDC
> >>> crashed and the krb5.conf is still pointing at a remote one. In that
> >>> case "malformed" replica's local host entry exist, but when such host
> >>> tries to get TGT, the AS-REQ goes to remote KDC from other master.
> >>>
> >>> However, if the admin had in the mean time cleaned up this host's
> >>> kerberos principals/keys, the crashed replica gets one of the following
> >>> errors:
> >>>
> >>> Client not found in Kerberos database
> >>> Client credentials have been revoked
> >>> Generic preauthentication failure
> >>>
> >>> These were printed out as errors during uninstall, but were actually
> >>> expected in situation like this. It is true that the code should check
> >>> and ignore these specific errors.
> >>
> >> Only the first id valid for your case, the others may be transient
> >> errors.
> >>
> >> Simo.
> >>
> >>
> > True, attaching updated patch. The other errors will now pop out in the
> > output and the warning will be displayed.
> >
> >
> >
> Bump for review.
> 

LGTM
Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Martin Babinsky]

On 01/14/2016 10:31 PM, Simo Sorce wrote:

On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote:

On 01/13/2016 10:31 AM, Martin Babinsky wrote:

On 01/07/2016 05:38 PM, Martin Babinsky wrote:

On 01/07/2016 05:37 PM, Martin Babinsky wrote:

https://fedorahosted.org/freeipa/ticket/5584


And the patch is here.




self-NACK, there may be a better way to handle this. I will do some
investigation and send updated patch.


Attaching updated patch.


A failure to obtain a tgt may be due to other reasons (for example the
KDC crashed), why are you trying to use this test ?
Isn't it sufficient to see there is no host entry in the directory ?

Simo.

There were some corner cases I encountered, mostly concerning a cleanup 
after unsuccessful replica promotion.


You may sometimes end up in a state where local DS is working, but KDC 
crashed and the krb5.conf is still pointing at a remote one. In that 
case "malformed" replica's local host entry exist, but when such host 
tries to get TGT, the AS-REQ goes to remote KDC from other master.


However, if the admin had in the mean time cleaned up this host's 
kerberos principals/keys, the crashed replica gets one of the following 
errors:


Client not found in Kerberos database
Client credentials have been revoked
Generic preauthentication failure

These were printed out as errors during uninstall, but were actually 
expected in situation like this. It is true that the code should check 
and ignore these specific errors.


--
Martin^3 Babinsky

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Simo Sorce]

On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote:
> On 01/14/2016 10:31 PM, Simo Sorce wrote:
> > On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote:
> >> On 01/13/2016 10:31 AM, Martin Babinsky wrote:
> >>> On 01/07/2016 05:38 PM, Martin Babinsky wrote:
>  On 01/07/2016 05:37 PM, Martin Babinsky wrote:
> > https://fedorahosted.org/freeipa/ticket/5584
> >
>  And the patch is here.
> 
> 
> 
> >>> self-NACK, there may be a better way to handle this. I will do some
> >>> investigation and send updated patch.
> >>>
> >> Attaching updated patch.
> >
> > A failure to obtain a tgt may be due to other reasons (for example the
> > KDC crashed), why are you trying to use this test ?
> > Isn't it sufficient to see there is no host entry in the directory ?
> >
> > Simo.
> >
> There were some corner cases I encountered, mostly concerning a cleanup 
> after unsuccessful replica promotion.
> 
> You may sometimes end up in a state where local DS is working, but KDC 
> crashed and the krb5.conf is still pointing at a remote one. In that 
> case "malformed" replica's local host entry exist, but when such host 
> tries to get TGT, the AS-REQ goes to remote KDC from other master.
> 
> However, if the admin had in the mean time cleaned up this host's 
> kerberos principals/keys, the crashed replica gets one of the following 
> errors:
> 
> Client not found in Kerberos database
> Client credentials have been revoked
> Generic preauthentication failure
> 
> These were printed out as errors during uninstall, but were actually 
> expected in situation like this. It is true that the code should check 
> and ignore these specific errors.

Only the first id valid for your case, the others may be transient
errors.

Simo.


-- 
Simo Sorce * Red Hat, Inc * New York

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Martin Babinsky]

On 01/15/2016 04:57 PM, Simo Sorce wrote:

On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote:

On 01/14/2016 10:31 PM, Simo Sorce wrote:

On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote:

On 01/13/2016 10:31 AM, Martin Babinsky wrote:

On 01/07/2016 05:38 PM, Martin Babinsky wrote:

On 01/07/2016 05:37 PM, Martin Babinsky wrote:

https://fedorahosted.org/freeipa/ticket/5584


And the patch is here.




self-NACK, there may be a better way to handle this. I will do some
investigation and send updated patch.


Attaching updated patch.


A failure to obtain a tgt may be due to other reasons (for example the
KDC crashed), why are you trying to use this test ?
Isn't it sufficient to see there is no host entry in the directory ?

Simo.


There were some corner cases I encountered, mostly concerning a cleanup
after unsuccessful replica promotion.

You may sometimes end up in a state where local DS is working, but KDC
crashed and the krb5.conf is still pointing at a remote one. In that
case "malformed" replica's local host entry exist, but when such host
tries to get TGT, the AS-REQ goes to remote KDC from other master.

However, if the admin had in the mean time cleaned up this host's
kerberos principals/keys, the crashed replica gets one of the following
errors:

Client not found in Kerberos database
Client credentials have been revoked
Generic preauthentication failure

These were printed out as errors during uninstall, but were actually
expected in situation like this. It is true that the code should check
and ignore these specific errors.


Only the first id valid for your case, the others may be transient
errors.

Simo.


True, attaching updated patch. The other errors will now pop out in the 
output and the warning will be displayed.


--
Martin^3 Babinsky
From 6517633c8b8019ad275e85c2273177a1275bdc62 Mon Sep 17 00:00:00 2001
From: Martin Babinsky 
Date: Thu, 7 Jan 2016 16:48:11 +0100
Subject: [PATCH] uninstallation: more robust check for master removal from
 topology

When uninstalling IPA master in domain level 1 topology, the code that checks
for correct removal from topology will now consider failures to lookup host
entry in local LDAP and to obtain host TGT as a sign that the master entry was
already removed.

https://fedorahosted.org/freeipa/ticket/5584
---
 ipalib/krb_utils.py |  1 +
 ipaserver/install/server/install.py | 40 +
 2 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/ipalib/krb_utils.py b/ipalib/krb_utils.py
index 0c4340c3f232135b64dafb6a675ffbcdd7ea59cd..b33e4b7c82cf08c68220531ebacca309117ad770 100644
--- a/ipalib/krb_utils.py
+++ b/ipalib/krb_utils.py
@@ -32,6 +32,7 @@ if six.PY3:
 # Kerberos error codes
 KRB5_CC_NOTFOUND= 2529639053 # Matching credential not found
 KRB5_FCC_NOFILE = 2529639107 # No credentials cache found
+KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN = 2529638918  # client not found in Kerberos db
 KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN = 2529638919 # Server not found in Kerberos database
 KRB5KRB_AP_ERR_TKT_EXPIRED  = 2529638944 # Ticket expired
 KRB5_FCC_PERM   = 2529639106 # Credentials cache permissions incorrect
diff --git a/ipaserver/install/server/install.py b/ipaserver/install/server/install.py
index 49e97eb667a322898acc3a064f4eae5381ded918..362b99f320a7e83ff0427924c41f3e26a42c3226 100644
--- a/ipaserver/install/server/install.py
+++ b/ipaserver/install/server/install.py
@@ -4,6 +4,7 @@
 
 from __future__ import print_function
 
+import gssapi
 import os
 import pickle
 import pwd
@@ -27,6 +28,7 @@ from ipaplatform import services
 from ipaplatform.paths import paths
 from ipaplatform.tasks import tasks
 from ipalib import api, create_api, constants, errors, x509
+from ipalib.krb_utils import KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN
 from ipalib.constants import CACERT
 from ipalib.util import validate_domain_name
 import ipaclient.ntpconf
@@ -291,20 +293,50 @@ def common_cleanup(func):
 
 
 def check_master_deleted(api, masters, interactive):
+"""
+Determine whether the IPA master was removed from the domain level 1
+topology. The function first tries to locally lookup the master host entry
+and fetches host prinicipal from DS. Then we attempt to acquire host TGT,
+contact the other masters one at a time and query for the existence of the
+host entry for our IPA master.
+
+:param api: instance of API object
+:param masters: list of masters to contact
+:param interactive: whether run in interactive mode. The user will be
+prompted for action if the removal status cannot be determined
+:return: True if the master is not part of the topology anymore as
+determined by the following conditions:
+* the host entry does not exist in local DS
+* request for host TGT fails due to missing/invalid/revoked creds
+* GSSAPI connection to remote DS fails on 

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Simo Sorce]

On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote:
> On 01/13/2016 10:31 AM, Martin Babinsky wrote:
> > On 01/07/2016 05:38 PM, Martin Babinsky wrote:
> >> On 01/07/2016 05:37 PM, Martin Babinsky wrote:
> >>> https://fedorahosted.org/freeipa/ticket/5584
> >>>
> >> And the patch is here.
> >>
> >>
> >>
> > self-NACK, there may be a better way to handle this. I will do some
> > investigation and send updated patch.
> >
> Attaching updated patch.

A failure to obtain a tgt may be due to other reasons (for example the
KDC crashed), why are you trying to use this test ?
Isn't it sufficient to see there is no host entry in the directory ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Martin Babinsky]

On 01/07/2016 05:38 PM, Martin Babinsky wrote:

On 01/07/2016 05:37 PM, Martin Babinsky wrote:

https://fedorahosted.org/freeipa/ticket/5584


And the patch is here.



self-NACK, there may be a better way to handle this. I will do some 
investigation and send updated patch.


--
Martin^3 Babinsky

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Martin Babinsky]

On 01/13/2016 10:31 AM, Martin Babinsky wrote:

On 01/07/2016 05:38 PM, Martin Babinsky wrote:

On 01/07/2016 05:37 PM, Martin Babinsky wrote:

https://fedorahosted.org/freeipa/ticket/5584


And the patch is here.




self-NACK, there may be a better way to handle this. I will do some
investigation and send updated patch.


Attaching updated patch.

--
Martin^3 Babinsky
From 0fe8f5e989f62c716f1de8159ca4d8c498106784 Mon Sep 17 00:00:00 2001
From: Martin Babinsky 
Date: Thu, 7 Jan 2016 16:48:11 +0100
Subject: [PATCH 1/3] uninstallation: more robust check for master removal from
 topology

When uninstalling IPA master in domain level 1 topology, the code that checks
for correct removal from topology will now consider failures to lookup host
entry in local LDAP and to obtain host TGT as a sign that the master entry was
already removed.

https://fedorahosted.org/freeipa/ticket/5584
---
 ipaserver/install/server/install.py | 37 +++--
 1 file changed, 31 insertions(+), 6 deletions(-)

diff --git a/ipaserver/install/server/install.py b/ipaserver/install/server/install.py
index 8a57886cd91bc4dbb06d30b457844499d3ff6cec..aa048cc6d05490ec38e4f2808e7874cd8312704b 100644
--- a/ipaserver/install/server/install.py
+++ b/ipaserver/install/server/install.py
@@ -4,6 +4,7 @@
 
 from __future__ import print_function
 
+import gssapi
 import os
 import pickle
 import pwd
@@ -291,26 +292,50 @@ def common_cleanup(func):
 
 
 def check_master_deleted(api, masters, interactive):
+"""
+Determine whether the IPA master was removed from the domain level 1
+topology. The function first tries to locally lookup the master host entry
+and fetches host prinicipal from DS. Then we attempt to acquire host TGT,
+contact the other masters one at a time and query for the existence of the
+host entry for our IPA master.
+
+:param api: instance of API object
+:param masters: list of masters to contact
+:param interactive: whether run in interactive mode. The user will be
+prompted for action if the removal status cannot be determined
+:return: True if the master is not part of the topology anymore as
+determined by the following conditions:
+* the host entry does not exist in local DS
+* we fail to get host TGT
+* GSSAPI connection to remote DS fails on invalid authentication
+* if we are the only master
+False otherwise
+"""
 try:
 host_princ = api.Command.host_show(
 api.env.host)['result']['krbprincipalname'][0]
-except Exception as e:
-root_logger.warning(
-"Failed to get host principal name: {0}".format(e)
+except errors.NotFound:
+root_logger.debug(
+"Host entry for {} already deleted".format(api.env.host)
 )
+return True
+except Exception as e:
+root_logger.warning("Failed to get host principal name: {0}".format(e))
 return False
 
 ccache_path = os.path.join('/', 'tmp', 'krb5cc_host')
 with ipautil.private_ccache(ccache_path):
+# attempt to get host TGT. Failure to do this indicates that the
+# master was removed from topology
 try:
 ipautil.kinit_keytab(host_princ, paths.KRB5_KEYTAB, ccache_path)
-except Exception as e:
-root_logger.error(
+except gssapi.exceptions.GSSError as e:
+root_logger.debug(
 "Kerberos authentication as '{0}' failed: {1}".format(
 host_princ, e
 )
 )
-return False
+return True
 
 last_server = True
 for master in masters:
-- 
2.5.0

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

[Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Martin Babinsky]

https://fedorahosted.org/freeipa/ticket/5584

--
Martin^3 Babinsky

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails

[Martin Babinsky]

On 01/07/2016 05:37 PM, Martin Babinsky wrote:

https://fedorahosted.org/freeipa/ticket/5584


And the patch is here.

--
Martin^3 Babinsky
From 43617fe3bbd4e72626bdf9f3c228c3585cc37d4b Mon Sep 17 00:00:00 2001
From: Martin Babinsky 
Date: Thu, 7 Jan 2016 16:48:11 +0100
Subject: [PATCH] consider IPA master removed from topology when request for
 host TGT fails

When uninstalling IPA master in domain level 1 topology, the code that checks
for correct removal from topology will now consider failure to obtain host TGT
as a sign that the master entry was already removed.

https://fedorahosted.org/freeipa/ticket/5584
---
 ipaserver/install/server/install.py | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/ipaserver/install/server/install.py b/ipaserver/install/server/install.py
index 8a57886cd91bc4dbb06d30b457844499d3ff6cec..aa7e071fb88115f6b7737468656b3fdb8d7ebc98 100644
--- a/ipaserver/install/server/install.py
+++ b/ipaserver/install/server/install.py
@@ -4,6 +4,7 @@
 
 from __future__ import print_function
 
+import gssapi
 import os
 import pickle
 import pwd
@@ -302,15 +303,17 @@ def check_master_deleted(api, masters, interactive):
 
 ccache_path = os.path.join('/', 'tmp', 'krb5cc_host')
 with ipautil.private_ccache(ccache_path):
+# attempt to get host TGT. Failure to do this indicates that the
+# master was removed from topology
 try:
 ipautil.kinit_keytab(host_princ, paths.KRB5_KEYTAB, ccache_path)
-except Exception as e:
-root_logger.error(
+except gssapi.exceptions.GSSError as e:
+root_logger.debug(
 "Kerberos authentication as '{0}' failed: {1}".format(
 host_princ, e
 )
 )
-return False
+return True
 
 last_server = True
 for master in masters:
-- 
2.5.0

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code