Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On 20.01.2016 15:45, Simo Sorce wrote: On Wed, 2016-01-20 at 09:42 +0100, Martin Babinsky wrote: On 01/15/2016 06:29 PM, Martin Babinsky wrote: On 01/15/2016 04:57 PM, Simo Sorce wrote: On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote: On 01/14/2016 10:31 PM, Simo Sorce wrote: On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote: On 01/13/2016 10:31 AM, Martin Babinsky wrote: On 01/07/2016 05:38 PM, Martin Babinsky wrote: On 01/07/2016 05:37 PM, Martin Babinsky wrote: https://fedorahosted.org/freeipa/ticket/5584 And the patch is here. self-NACK, there may be a better way to handle this. I will do some investigation and send updated patch. Attaching updated patch. A failure to obtain a tgt may be due to other reasons (for example the KDC crashed), why are you trying to use this test ? Isn't it sufficient to see there is no host entry in the directory ? Simo. There were some corner cases I encountered, mostly concerning a cleanup after unsuccessful replica promotion. You may sometimes end up in a state where local DS is working, but KDC crashed and the krb5.conf is still pointing at a remote one. In that case "malformed" replica's local host entry exist, but when such host tries to get TGT, the AS-REQ goes to remote KDC from other master. However, if the admin had in the mean time cleaned up this host's kerberos principals/keys, the crashed replica gets one of the following errors: Client not found in Kerberos database Client credentials have been revoked Generic preauthentication failure These were printed out as errors during uninstall, but were actually expected in situation like this. It is true that the code should check and ignore these specific errors. Only the first id valid for your case, the others may be transient errors. Simo. True, attaching updated patch. The other errors will now pop out in the output and the warning will be displayed. Bump for review. LGTM Simo. ACK Pushed to: master: d726da3ba20283ffdc1d384dfedf8e6a732dc3d7 ipa-4-3: 4f0266f925207ca705b45287744b3e609d841cc6 -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On Wed, 2016-01-20 at 09:42 +0100, Martin Babinsky wrote: > On 01/15/2016 06:29 PM, Martin Babinsky wrote: > > On 01/15/2016 04:57 PM, Simo Sorce wrote: > >> On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote: > >>> On 01/14/2016 10:31 PM, Simo Sorce wrote: > On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote: > > On 01/13/2016 10:31 AM, Martin Babinsky wrote: > >> On 01/07/2016 05:38 PM, Martin Babinsky wrote: > >>> On 01/07/2016 05:37 PM, Martin Babinsky wrote: > https://fedorahosted.org/freeipa/ticket/5584 > > >>> And the patch is here. > >>> > >>> > >>> > >> self-NACK, there may be a better way to handle this. I will do some > >> investigation and send updated patch. > >> > > Attaching updated patch. > > A failure to obtain a tgt may be due to other reasons (for example the > KDC crashed), why are you trying to use this test ? > Isn't it sufficient to see there is no host entry in the directory ? > > Simo. > > >>> There were some corner cases I encountered, mostly concerning a cleanup > >>> after unsuccessful replica promotion. > >>> > >>> You may sometimes end up in a state where local DS is working, but KDC > >>> crashed and the krb5.conf is still pointing at a remote one. In that > >>> case "malformed" replica's local host entry exist, but when such host > >>> tries to get TGT, the AS-REQ goes to remote KDC from other master. > >>> > >>> However, if the admin had in the mean time cleaned up this host's > >>> kerberos principals/keys, the crashed replica gets one of the following > >>> errors: > >>> > >>> Client not found in Kerberos database > >>> Client credentials have been revoked > >>> Generic preauthentication failure > >>> > >>> These were printed out as errors during uninstall, but were actually > >>> expected in situation like this. It is true that the code should check > >>> and ignore these specific errors. > >> > >> Only the first id valid for your case, the others may be transient > >> errors. > >> > >> Simo. > >> > >> > > True, attaching updated patch. The other errors will now pop out in the > > output and the warning will be displayed. > > > > > > > Bump for review. > LGTM Simo. -- Simo Sorce * Red Hat, Inc * New York -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On 01/15/2016 06:29 PM, Martin Babinsky wrote: On 01/15/2016 04:57 PM, Simo Sorce wrote: On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote: On 01/14/2016 10:31 PM, Simo Sorce wrote: On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote: On 01/13/2016 10:31 AM, Martin Babinsky wrote: On 01/07/2016 05:38 PM, Martin Babinsky wrote: On 01/07/2016 05:37 PM, Martin Babinsky wrote: https://fedorahosted.org/freeipa/ticket/5584 And the patch is here. self-NACK, there may be a better way to handle this. I will do some investigation and send updated patch. Attaching updated patch. A failure to obtain a tgt may be due to other reasons (for example the KDC crashed), why are you trying to use this test ? Isn't it sufficient to see there is no host entry in the directory ? Simo. There were some corner cases I encountered, mostly concerning a cleanup after unsuccessful replica promotion. You may sometimes end up in a state where local DS is working, but KDC crashed and the krb5.conf is still pointing at a remote one. In that case "malformed" replica's local host entry exist, but when such host tries to get TGT, the AS-REQ goes to remote KDC from other master. However, if the admin had in the mean time cleaned up this host's kerberos principals/keys, the crashed replica gets one of the following errors: Client not found in Kerberos database Client credentials have been revoked Generic preauthentication failure These were printed out as errors during uninstall, but were actually expected in situation like this. It is true that the code should check and ignore these specific errors. Only the first id valid for your case, the others may be transient errors. Simo. True, attaching updated patch. The other errors will now pop out in the output and the warning will be displayed. Bump for review. -- Martin^3 Babinsky -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On 01/15/2016 04:57 PM, Simo Sorce wrote: On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote: On 01/14/2016 10:31 PM, Simo Sorce wrote: On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote: On 01/13/2016 10:31 AM, Martin Babinsky wrote: On 01/07/2016 05:38 PM, Martin Babinsky wrote: On 01/07/2016 05:37 PM, Martin Babinsky wrote: https://fedorahosted.org/freeipa/ticket/5584 And the patch is here. self-NACK, there may be a better way to handle this. I will do some investigation and send updated patch. Attaching updated patch. A failure to obtain a tgt may be due to other reasons (for example the KDC crashed), why are you trying to use this test ? Isn't it sufficient to see there is no host entry in the directory ? Simo. There were some corner cases I encountered, mostly concerning a cleanup after unsuccessful replica promotion. You may sometimes end up in a state where local DS is working, but KDC crashed and the krb5.conf is still pointing at a remote one. In that case "malformed" replica's local host entry exist, but when such host tries to get TGT, the AS-REQ goes to remote KDC from other master. However, if the admin had in the mean time cleaned up this host's kerberos principals/keys, the crashed replica gets one of the following errors: Client not found in Kerberos database Client credentials have been revoked Generic preauthentication failure These were printed out as errors during uninstall, but were actually expected in situation like this. It is true that the code should check and ignore these specific errors. Only the first id valid for your case, the others may be transient errors. Simo. True, attaching updated patch. The other errors will now pop out in the output and the warning will be displayed. -- Martin^3 Babinsky From 6517633c8b8019ad275e85c2273177a1275bdc62 Mon Sep 17 00:00:00 2001 From: Martin Babinsky Date: Thu, 7 Jan 2016 16:48:11 +0100 Subject: [PATCH] uninstallation: more robust check for master removal from topology When uninstalling IPA master in domain level 1 topology, the code that checks for correct removal from topology will now consider failures to lookup host entry in local LDAP and to obtain host TGT as a sign that the master entry was already removed. https://fedorahosted.org/freeipa/ticket/5584 --- ipalib/krb_utils.py | 1 + ipaserver/install/server/install.py | 40 + 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/ipalib/krb_utils.py b/ipalib/krb_utils.py index 0c4340c3f232135b64dafb6a675ffbcdd7ea59cd..b33e4b7c82cf08c68220531ebacca309117ad770 100644 --- a/ipalib/krb_utils.py +++ b/ipalib/krb_utils.py @@ -32,6 +32,7 @@ if six.PY3: # Kerberos error codes KRB5_CC_NOTFOUND= 2529639053 # Matching credential not found KRB5_FCC_NOFILE = 2529639107 # No credentials cache found +KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN = 2529638918 # client not found in Kerberos db KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN = 2529638919 # Server not found in Kerberos database KRB5KRB_AP_ERR_TKT_EXPIRED = 2529638944 # Ticket expired KRB5_FCC_PERM = 2529639106 # Credentials cache permissions incorrect diff --git a/ipaserver/install/server/install.py b/ipaserver/install/server/install.py index 49e97eb667a322898acc3a064f4eae5381ded918..362b99f320a7e83ff0427924c41f3e26a42c3226 100644 --- a/ipaserver/install/server/install.py +++ b/ipaserver/install/server/install.py @@ -4,6 +4,7 @@ from __future__ import print_function +import gssapi import os import pickle import pwd @@ -27,6 +28,7 @@ from ipaplatform import services from ipaplatform.paths import paths from ipaplatform.tasks import tasks from ipalib import api, create_api, constants, errors, x509 +from ipalib.krb_utils import KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN from ipalib.constants import CACERT from ipalib.util import validate_domain_name import ipaclient.ntpconf @@ -291,20 +293,50 @@ def common_cleanup(func): def check_master_deleted(api, masters, interactive): +""" +Determine whether the IPA master was removed from the domain level 1 +topology. The function first tries to locally lookup the master host entry +and fetches host prinicipal from DS. Then we attempt to acquire host TGT, +contact the other masters one at a time and query for the existence of the +host entry for our IPA master. + +:param api: instance of API object +:param masters: list of masters to contact +:param interactive: whether run in interactive mode. The user will be +prompted for action if the removal status cannot be determined +:return: True if the master is not part of the topology anymore as +determined by the following conditions: +* the host entry does not exist in local DS +* request for host TGT fails due to missing/invalid/revoked creds +* GSSAPI connection to remote DS fails on invalid authentication +
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On Fri, 2016-01-15 at 13:35 +0100, Martin Babinsky wrote: > On 01/14/2016 10:31 PM, Simo Sorce wrote: > > On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote: > >> On 01/13/2016 10:31 AM, Martin Babinsky wrote: > >>> On 01/07/2016 05:38 PM, Martin Babinsky wrote: > On 01/07/2016 05:37 PM, Martin Babinsky wrote: > > https://fedorahosted.org/freeipa/ticket/5584 > > > And the patch is here. > > > > >>> self-NACK, there may be a better way to handle this. I will do some > >>> investigation and send updated patch. > >>> > >> Attaching updated patch. > > > > A failure to obtain a tgt may be due to other reasons (for example the > > KDC crashed), why are you trying to use this test ? > > Isn't it sufficient to see there is no host entry in the directory ? > > > > Simo. > > > There were some corner cases I encountered, mostly concerning a cleanup > after unsuccessful replica promotion. > > You may sometimes end up in a state where local DS is working, but KDC > crashed and the krb5.conf is still pointing at a remote one. In that > case "malformed" replica's local host entry exist, but when such host > tries to get TGT, the AS-REQ goes to remote KDC from other master. > > However, if the admin had in the mean time cleaned up this host's > kerberos principals/keys, the crashed replica gets one of the following > errors: > > Client not found in Kerberos database > Client credentials have been revoked > Generic preauthentication failure > > These were printed out as errors during uninstall, but were actually > expected in situation like this. It is true that the code should check > and ignore these specific errors. Only the first id valid for your case, the others may be transient errors. Simo. -- Simo Sorce * Red Hat, Inc * New York -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On 01/14/2016 10:31 PM, Simo Sorce wrote: On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote: On 01/13/2016 10:31 AM, Martin Babinsky wrote: On 01/07/2016 05:38 PM, Martin Babinsky wrote: On 01/07/2016 05:37 PM, Martin Babinsky wrote: https://fedorahosted.org/freeipa/ticket/5584 And the patch is here. self-NACK, there may be a better way to handle this. I will do some investigation and send updated patch. Attaching updated patch. A failure to obtain a tgt may be due to other reasons (for example the KDC crashed), why are you trying to use this test ? Isn't it sufficient to see there is no host entry in the directory ? Simo. There were some corner cases I encountered, mostly concerning a cleanup after unsuccessful replica promotion. You may sometimes end up in a state where local DS is working, but KDC crashed and the krb5.conf is still pointing at a remote one. In that case "malformed" replica's local host entry exist, but when such host tries to get TGT, the AS-REQ goes to remote KDC from other master. However, if the admin had in the mean time cleaned up this host's kerberos principals/keys, the crashed replica gets one of the following errors: Client not found in Kerberos database Client credentials have been revoked Generic preauthentication failure These were printed out as errors during uninstall, but were actually expected in situation like this. It is true that the code should check and ignore these specific errors. -- Martin^3 Babinsky -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On Wed, 2016-01-13 at 17:31 +0100, Martin Babinsky wrote: > On 01/13/2016 10:31 AM, Martin Babinsky wrote: > > On 01/07/2016 05:38 PM, Martin Babinsky wrote: > >> On 01/07/2016 05:37 PM, Martin Babinsky wrote: > >>> https://fedorahosted.org/freeipa/ticket/5584 > >>> > >> And the patch is here. > >> > >> > >> > > self-NACK, there may be a better way to handle this. I will do some > > investigation and send updated patch. > > > Attaching updated patch. A failure to obtain a tgt may be due to other reasons (for example the KDC crashed), why are you trying to use this test ? Isn't it sufficient to see there is no host entry in the directory ? Simo. -- Simo Sorce * Red Hat, Inc * New York -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On 01/13/2016 10:31 AM, Martin Babinsky wrote: On 01/07/2016 05:38 PM, Martin Babinsky wrote: On 01/07/2016 05:37 PM, Martin Babinsky wrote: https://fedorahosted.org/freeipa/ticket/5584 And the patch is here. self-NACK, there may be a better way to handle this. I will do some investigation and send updated patch. Attaching updated patch. -- Martin^3 Babinsky From 0fe8f5e989f62c716f1de8159ca4d8c498106784 Mon Sep 17 00:00:00 2001 From: Martin Babinsky Date: Thu, 7 Jan 2016 16:48:11 +0100 Subject: [PATCH 1/3] uninstallation: more robust check for master removal from topology When uninstalling IPA master in domain level 1 topology, the code that checks for correct removal from topology will now consider failures to lookup host entry in local LDAP and to obtain host TGT as a sign that the master entry was already removed. https://fedorahosted.org/freeipa/ticket/5584 --- ipaserver/install/server/install.py | 37 +++-- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/ipaserver/install/server/install.py b/ipaserver/install/server/install.py index 8a57886cd91bc4dbb06d30b457844499d3ff6cec..aa048cc6d05490ec38e4f2808e7874cd8312704b 100644 --- a/ipaserver/install/server/install.py +++ b/ipaserver/install/server/install.py @@ -4,6 +4,7 @@ from __future__ import print_function +import gssapi import os import pickle import pwd @@ -291,26 +292,50 @@ def common_cleanup(func): def check_master_deleted(api, masters, interactive): +""" +Determine whether the IPA master was removed from the domain level 1 +topology. The function first tries to locally lookup the master host entry +and fetches host prinicipal from DS. Then we attempt to acquire host TGT, +contact the other masters one at a time and query for the existence of the +host entry for our IPA master. + +:param api: instance of API object +:param masters: list of masters to contact +:param interactive: whether run in interactive mode. The user will be +prompted for action if the removal status cannot be determined +:return: True if the master is not part of the topology anymore as +determined by the following conditions: +* the host entry does not exist in local DS +* we fail to get host TGT +* GSSAPI connection to remote DS fails on invalid authentication +* if we are the only master +False otherwise +""" try: host_princ = api.Command.host_show( api.env.host)['result']['krbprincipalname'][0] -except Exception as e: -root_logger.warning( -"Failed to get host principal name: {0}".format(e) +except errors.NotFound: +root_logger.debug( +"Host entry for {} already deleted".format(api.env.host) ) +return True +except Exception as e: +root_logger.warning("Failed to get host principal name: {0}".format(e)) return False ccache_path = os.path.join('/', 'tmp', 'krb5cc_host') with ipautil.private_ccache(ccache_path): +# attempt to get host TGT. Failure to do this indicates that the +# master was removed from topology try: ipautil.kinit_keytab(host_princ, paths.KRB5_KEYTAB, ccache_path) -except Exception as e: -root_logger.error( +except gssapi.exceptions.GSSError as e: +root_logger.debug( "Kerberos authentication as '{0}' failed: {1}".format( host_princ, e ) ) -return False +return True last_server = True for master in masters: -- 2.5.0 -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On 01/07/2016 05:38 PM, Martin Babinsky wrote: On 01/07/2016 05:37 PM, Martin Babinsky wrote: https://fedorahosted.org/freeipa/ticket/5584 And the patch is here. self-NACK, there may be a better way to handle this. I will do some investigation and send updated patch. -- Martin^3 Babinsky -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0121] consider IPA master removed from topology when request for host TGT fails
On 01/07/2016 05:37 PM, Martin Babinsky wrote: https://fedorahosted.org/freeipa/ticket/5584 And the patch is here. -- Martin^3 Babinsky From 43617fe3bbd4e72626bdf9f3c228c3585cc37d4b Mon Sep 17 00:00:00 2001 From: Martin Babinsky Date: Thu, 7 Jan 2016 16:48:11 +0100 Subject: [PATCH] consider IPA master removed from topology when request for host TGT fails When uninstalling IPA master in domain level 1 topology, the code that checks for correct removal from topology will now consider failure to obtain host TGT as a sign that the master entry was already removed. https://fedorahosted.org/freeipa/ticket/5584 --- ipaserver/install/server/install.py | 9 ++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/ipaserver/install/server/install.py b/ipaserver/install/server/install.py index 8a57886cd91bc4dbb06d30b457844499d3ff6cec..aa7e071fb88115f6b7737468656b3fdb8d7ebc98 100644 --- a/ipaserver/install/server/install.py +++ b/ipaserver/install/server/install.py @@ -4,6 +4,7 @@ from __future__ import print_function +import gssapi import os import pickle import pwd @@ -302,15 +303,17 @@ def check_master_deleted(api, masters, interactive): ccache_path = os.path.join('/', 'tmp', 'krb5cc_host') with ipautil.private_ccache(ccache_path): +# attempt to get host TGT. Failure to do this indicates that the +# master was removed from topology try: ipautil.kinit_keytab(host_princ, paths.KRB5_KEYTAB, ccache_path) -except Exception as e: -root_logger.error( +except gssapi.exceptions.GSSError as e: +root_logger.debug( "Kerberos authentication as '{0}' failed: {1}".format( host_princ, e ) ) -return False +return True last_server = True for master in masters: -- 2.5.0 -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code