This is a WIP patch which moves the `ipa-replica-manage del` subcommand to the 'server-del' API method and exposes it as CLI command[1]. A CI test suite is also included.

There are some issues with the patch I would like to discuss in more detail on the list:

1.) In the original subcommand there was a lot of output (mostly print statements) during all stages of master removal. I have tried to port these as messages to the command which results in quite voluminous response sent back to the frontend. Should we try to reduce the output?

2.) In the original discussion[2] we assumed that the cleanup part would me a separate API method called during server_del postcallback. However since the two objects ended up sharing a lot of state (e.g. topology state from pre-callback, messages) i have merged it to server-del. That makes the code rather unwieldy but I found it difficult to keep the two entities separate without some hacking around framework capabilities

3.) since actions in post-callback require a knowledge about topology state gathered in pre-callback, I had to store some information in the command's context. Sorry about that, if you know about some way to circumvent me, let me know.

4.) The master can not remove itself. I have implemented an ad-hoc forwarding of the request to other master that can do the job. Is this okay?

5.) Since the original behavior of 'chekc_deleted_segments' was kept, the code can sometimes hang waiting for removal of some segments, especially during forced removal in wonky topologies. This can cause gateway timeout in JSON-RPC call and report false positive error back to the user. This makes a large part of 'TestServerDel' suite to fail. How should we handle this? Should we keep the original behavior?

6.) There are some in-place imports of server-side code in some places. I'm not very happy about it and would be glad if we can agree on a way to fix this.


[1] https://fedorahosted.org/freeipa/ticket/5588
[2] https://www.redhat.com/archives/freeipa-devel/2016-March/msg00335.html

--
Martin^3 Babinsky
From e807b266a126e9708329871362aa52a7e15f4183 Mon Sep 17 00:00:00 2001
From: Martin Babinsky <mbabi...@redhat.com>
Date: Fri, 8 Apr 2016 16:29:04 +0200
Subject: [PATCH] integration test suite for server-del

https://fedorahosted.org/freeipa/ticket/5588
---
 ipatests/test_integration/tasks.py           |  32 ++-
 ipatests/test_integration/test_caless.py     |  11 +-
 ipatests/test_integration/test_server_del.py | 307 +++++++++++++++++++++++++++
 3 files changed, 335 insertions(+), 15 deletions(-)
 create mode 100644 ipatests/test_integration/test_server_del.py

diff --git a/ipatests/test_integration/tasks.py b/ipatests/test_integration/tasks.py
index 70f4fa7fd96f9d993332996f087577d1c88f828e..f3be216ee75a5d25e0722cb21ac64ad43318b623 100644
--- a/ipatests/test_integration/tasks.py
+++ b/ipatests/test_integration/tasks.py
@@ -57,10 +57,10 @@ def check_arguments_are(slice, instanceof):
     and third arguments are integers
     """
     def wrapper(func):
-        def wrapped(*args):
+        def wrapped(*args, **kwargs):
             for i in args[slice[0]:slice[1]]:
                 assert isinstance(i, instanceof), "Wrong type: %s: %s" % (i, type(i))
-            return func(*args)
+            return func(*args, **kwargs)
         return wrapped
     return wrapper
 
@@ -721,7 +721,7 @@ def clean_replication_agreement(master, replica):
 
 
 @check_arguments_are((0, 3), Host)
-def create_segment(master, leftnode, rightnode):
+def create_segment(master, leftnode, rightnode, suffix=DOMAIN_SUFFIX_NAME):
     """
     creates a topology segment. The first argument is a node to run the command
     :returns: a hash object containing segment's name, leftnode, rightnode
@@ -731,7 +731,7 @@ def create_segment(master, leftnode, rightnode):
     lefthost = leftnode.hostname
     righthost = rightnode.hostname
     segment_name = "%s-to-%s" % (lefthost, righthost)
-    result = master.run_command(["ipa", "topologysegment-add", DOMAIN_SUFFIX_NAME,
+    result = master.run_command(["ipa", "topologysegment-add", suffix,
                                  segment_name,
                                  "--leftnode=%s" % lefthost,
                                  "--rightnode=%s" % righthost], raiseonerr=False)
@@ -743,7 +743,7 @@ def create_segment(master, leftnode, rightnode):
         return {}, result.stderr_text
 
 
-def destroy_segment(master, segment_name):
+def destroy_segment(master, segment_name, suffix=DOMAIN_SUFFIX_NAME):
     """
     Destroys topology segment.
     :param master: reference to master object of class Host
@@ -753,7 +753,7 @@ def destroy_segment(master, segment_name):
     kinit_admin(master)
     command = ["ipa",
                "topologysegment-del",
-               DOMAIN_SUFFIX_NAME,
+               suffix,
                segment_name]
     result = master.run_command(command, raiseonerr=False)
     return result.returncode, result.stderr_text
@@ -1181,3 +1181,23 @@ def replicas_cleanup(func):
                                             "host-del",
                                             host.hostname], raiseonerr=False)
     return wrapped
+
+
+def run_server_del(host, server_to_delete, force_removal=False, cleanup=False):
+    kinit_admin(host)
+    args = ['ipa', 'server-del', server_to_delete]
+    if force_removal:
+        args.append('--force-removal')
+    if cleanup:
+        args.append('--cleanup')
+
+    return host.run_command(args, raiseonerr=False)
+
+
+def assert_error(result, stderr_text, returncode=None):
+    "Assert that `result` command failed and its stderr contains `stderr_text`"
+    assert stderr_text in result.stderr_text, result.stderr_text
+    if returncode:
+        assert result.returncode == returncode
+    else:
+        assert result.returncode > 0
diff --git a/ipatests/test_integration/test_caless.py b/ipatests/test_integration/test_caless.py
index fdc4fc8efe73631e9ab03f3b9019444f7d7e09ec..667e2b3b1d91f967b32fabdb7e472886bbdf79d7 100644
--- a/ipatests/test_integration/test_caless.py
+++ b/ipatests/test_integration/test_caless.py
@@ -35,6 +35,8 @@ from ipatests.test_integration import tasks
 
 _DEFAULT = object()
 
+assert_error = tasks.assert_error
+
 
 def get_install_stdin(cert_passwords=()):
     lines = [
@@ -56,15 +58,6 @@ def get_replica_prepare_stdin(cert_passwords=()):
     return '\n'.join(lines + [''])
 
 
-def assert_error(result, stderr_text, returncode=None):
-    "Assert that `result` command failed and its stderr contains `stderr_text`"
-    assert stderr_text in result.stderr_text, result.stderr_text
-    if returncode:
-        assert result.returncode == returncode
-    else:
-        assert result.returncode > 0
-
-
 class CALessBase(IntegrationTest):
     @classmethod
     def install(cls, mh):
diff --git a/ipatests/test_integration/test_server_del.py b/ipatests/test_integration/test_server_del.py
new file mode 100644
index 0000000000000000000000000000000000000000..1ee4fb854a0e42db279fdb21f0747bd72d3c9d82
--- /dev/null
+++ b/ipatests/test_integration/test_server_del.py
@@ -0,0 +1,307 @@
+#
+# Copyright (C) 2016  FreeIPA Contributors see COPYING for license
+#
+
+from itertools import permutations
+
+from ipatests.test_integration.base import IntegrationTest
+from ipatests.test_integration import tasks
+from ipalib.constants import DOMAIN_LEVEL_1, DOMAIN_SUFFIX_NAME, CA_SUFFIX_NAME
+
+REMOVAL_ERR_TEMPLATE = ("Removal of '{hostname}' leads to disconnected "
+                        "topology in suffix '{suffix}'")
+
+
+def check_master_removal(host, hostname_to_remove,
+                         force_removal=False,
+                         suffixes=(DOMAIN_SUFFIX_NAME,)):
+    result = tasks.run_server_del(host, hostname_to_remove,
+                                  force_removal=True)
+    assert result.returncode == 0
+    if force_removal:
+        assert ("Forcing removal of {hostname}".format(
+            hostname=hostname_to_remove) in result.stderr_text)
+    tasks.assert_error(
+        host.run_command(
+            ['ipa', 'server-show', hostname_to_remove], raiseonerr=False
+        ),
+        "Server {} not found".format(hostname_to_remove),
+        returncode=1
+    )
+    for suffix in suffixes:
+        segments = host.run_command(
+            ['ipa', 'topologysegment-find', suffix]
+        )
+        assert hostname_to_remove not in segments.stdout_text
+
+
+def check_removal_disconnects_topology(
+        host, hostname_to_remove,
+        affected_suffixes=(DOMAIN_SUFFIX_NAME,)):
+    result = tasks.run_server_del(host, hostname_to_remove)
+    assert len(affected_suffixes) <= 2
+
+    err_messages_by_suffix = {
+        CA_SUFFIX_NAME: REMOVAL_ERR_TEMPLATE.format(
+            hostname=hostname_to_remove,
+            suffix=CA_SUFFIX_NAME
+        ),
+        DOMAIN_SUFFIX_NAME: REMOVAL_ERR_TEMPLATE.format(
+            hostname=hostname_to_remove,
+            suffix=DOMAIN_SUFFIX_NAME
+        )
+    }
+
+    for suffix in err_messages_by_suffix:
+        if suffix in affected_suffixes:
+            tasks.assert_error(
+                result, err_messages_by_suffix[suffix], returncode=1)
+        else:
+            assert err_messages_by_suffix[suffix] not in result.stderr_text
+
+
+class ServerDelBase(IntegrationTest):
+    num_replicas = 2
+    num_clients = 1
+    domain_level = DOMAIN_LEVEL_1
+    topology = 'star'
+
+    @classmethod
+    def install(cls, mh):
+        super(ServerDelBase, cls).install(mh)
+
+        cls.client = cls.clients[0]
+        cls.replica1 = cls.replicas[0]
+        cls.replica2 = cls.replicas[1]
+
+
+class TestServerDel(ServerDelBase):
+
+    @classmethod
+    def install(cls, mh):
+        super(TestServerDel, cls).install(mh)
+        # prepare topologysegments for negative test cases
+        # it should look like this for DOMAIN_SUFFIX_NAME:
+        #             master
+        #            /
+        #           /
+        #          /
+        #   replica1------- replica2
+        # and like this for CA_SUFFIX_NAME
+        #             master
+        #                  \
+        #                   \
+        #                    \
+        #   replica1------- replica2
+
+        tasks.create_segment(cls.client, cls.replica1, cls.replica2)
+        tasks.create_segment(cls.client, cls.replica1, cls.replica2,
+                             suffix=CA_SUFFIX_NAME)
+
+        # try to delete all relevant segment connecting master and replica1/2
+        segment_name_fmt = '{p[0].hostname}-to-{p[1].hostname}'
+        for domain_pair in permutations((cls.master, cls.replica2)):
+            tasks.destroy_segment(
+                cls.client, segment_name_fmt.format(p=domain_pair))
+
+        for ca_pair in permutations((cls.master, cls.replica1)):
+            tasks.destroy_segment(
+                cls.client, segment_name_fmt.format(p=ca_pair),
+                suffix=CA_SUFFIX_NAME)
+
+    def test_removal_of_nonexistent_master_raises_error(self):
+        """
+        tests that removal of non-existent master raises an error
+        """
+        hostname = u'bogus-master.bogus.domain'
+        err_message = "{}: server not found".format(hostname)
+        tasks.assert_error(
+            tasks.run_server_del(self.client, hostname),
+            err_message,
+            returncode=2
+        )
+
+    def test_forced_removal_of_nonexistent_master_raises_error(self):
+        """
+        tests that forced removal of non-existent master raises an error
+        """
+        hostname = u'bogus-master.bogus.domain'
+        err_message = "{}: server not found".format(hostname)
+        tasks.assert_error(
+            tasks.run_server_del(self.client, hostname, force_removal=True),
+            err_message,
+            returncode=2
+        )
+
+    def test_cleanup_of_nonexistent_master(self):
+        """
+        tests that cleanup of non-existent master does not raise an error
+        """
+        hostname = u'bogus-master.bogus.domain'
+        result = tasks.run_server_del(self.client, hostname, cleanup=True)
+        assert result.returncode == 0
+        assert ("Master already deleted, proceeding with cleanup" in
+                result.stderr_text)
+
+    def test_removal_of_replica1_disconnects_domain_topology(self):
+        """
+        tests that given the used topology, attempted removal of replica1 fails
+        with disconnected DOMAIN topology but not CA
+        """
+
+        check_removal_disconnects_topology(
+            self.client,
+            self.replica1.hostname,
+            affected_suffixes=(DOMAIN_SUFFIX_NAME,)
+        )
+
+    def test_removal_of_replica2_disconnects_ca_topology(self):
+        """
+        tests that given the used topology, attempted removal of replica2 fails
+        with disconnected CA topology but not DOMAIN
+        """
+
+        check_removal_disconnects_topology(
+            self.client,
+            self.replica2.hostname,
+            affected_suffixes=(CA_SUFFIX_NAME,)
+        )
+
+    def test_forced_removal_of_replica1(self):
+        """
+        tests that forced removal of replica1 indeed bypasses all checks and
+        destroys the master for good
+        """
+        check_master_removal(
+            self.client,
+            self.replica1.hostname,
+            suffixes=(DOMAIN_SUFFIX_NAME,),
+            force_removal=True
+        )
+
+        # reinstall the replica
+        tasks.uninstall_master(self.replica1)
+        tasks.install_replica(self.master, self.replica1, setup_ca=True)
+
+    def test_forced_removal_of_replica2(self):
+        """
+        tests that forced removal of replica2 indeed bypasses all checks and
+        destroys the master for good
+        """
+        check_master_removal(
+            self.client,
+            self.replica2.hostname,
+            suffixes=(CA_SUFFIX_NAME,),
+            force_removal=True
+        )
+
+        # reinstall the replica
+        tasks.uninstall_master(self.replica2)
+        tasks.install_replica(self.master, self.replica2, setup_ca=True)
+
+    def test_removal_of_master_disconnects_both_topologies(self):
+        """
+        tests that master removal will now raise errors in both suffixes.
+        the master should be able to forward the request to other master since
+        it cannot remove itself. If not the test will fail since the client
+        forwards the request directly to master in this setup
+        """
+        check_removal_disconnects_topology(
+            self.client,
+            self.master.hostname,
+            affected_suffixes=(CA_SUFFIX_NAME, DOMAIN_SUFFIX_NAME)
+        )
+
+    def test_removal_of_replica1(self):
+        """
+        tests the removal of replica1 which should now pass without errors
+        """
+        check_master_removal(
+            self.client,
+            self.replica1.hostname,
+            suffixes=(CA_SUFFIX_NAME, DOMAIN_SUFFIX_NAME)
+        )
+
+    def test_removal_of_replica2(self):
+        """
+        tests the removal of replica2 which should now pass without errors
+        """
+        check_master_removal(
+            self.client,
+            self.replica2.hostname,
+            suffixes=(CA_SUFFIX_NAME, DOMAIN_SUFFIX_NAME)
+        )
+
+
+class TestLastServices(ServerDelBase):
+    """
+    Test the checks for last services during server-del and their bypassing
+    using when forcing the removal
+    """
+    num_replicas = 1
+    domain_level = DOMAIN_LEVEL_1
+    topology = 'line'
+
+    @classmethod
+    def install(cls, mh):
+        tasks.install_topo(
+            cls.topology, cls.master, cls.replicas, [],
+            domain_level=cls.domain_level, setup_replica_cas=False)
+
+    def test_removal_of_master_raises_error_about_last_ca(self):
+        """
+        test that removal of master fails on the last
+        """
+        tasks.assert_error(
+            tasks.run_server_del(self.replicas[0], self.master.hostname),
+            "Deleting this server is not allowed as it would leave your "
+            "installation without a CA.",
+            1
+        )
+
+    def test_install_ca_on_replica1(self):
+        """
+        Install CA on replica so that we can test DNS-related checks
+        """
+        tasks.install_ca(self.replicas[0], domain_level=self.domain_level)
+
+    def test_removal_of_master_raises_error_about_last_dns(self):
+        """
+        Now server-del should complain about the removal of last DNS server
+        """
+        tasks.assert_error(
+            tasks.run_server_del(self.replicas[0], self.master.hostname),
+            "Deleting this server will leave your installation "
+            "without a DNS.",
+            1
+        )
+
+    def test_install_dns_on_replica1_and_dnssec_on_master(self):
+        """
+        install DNS server on replica and DNSSec on master
+        """
+        tasks.install_dns(self.replicas[0])
+        args = [
+            "ipa-dns-install",
+            "--dnssec-master",
+            "--forwarder", self.master.config.dns_forwarder,
+            "-U",
+        ]
+        self.master.run_command(args)
+
+    def test_removal_of_master_raises_errorabout_dnssec(self):
+        tasks.assert_error(
+            tasks.run_server_del(self.replicas[0], self.master.hostname),
+            "Replica is active DNSSEC key master. Uninstall "
+            "could break your DNS system. Please disable or replace "
+            "DNSSEC key master first.",
+            1
+        )
+
+    def test_forced_removal_of_master(self):
+        """
+        Tests that we can still force remove the master
+        """
+        check_master_removal(
+            self.replicas[0], self.master.hostname, force_removal=True,
+            suffixes=(DOMAIN_SUFFIX_NAME, CA_SUFFIX_NAME))
-- 
2.5.5

From b9613dbb0c419f38023dc2534aa1dfa27242e741 Mon Sep 17 00:00:00 2001
From: Martin Babinsky <mbabi...@redhat.com>
Date: Thu, 24 Mar 2016 13:43:07 +0100
Subject: [PATCH 1/2] server-del: perform full master removal in managed
 topology

This patch implements most of the del_master_managed() functionality as a part
of `server-del` command.

`server-del` nows performs these actions:
* check topology connectivity
* check that at least one CA/DNS server and DNSSec masters are left
  after removal
* remove master entry from LDAP
* check that all segments pointing to the master were removed
* cleanup all leftover LDAP entries exposing information about the master
* cleanup leftover master DNS records

`server-del` now accepts the following options:
* `--cleanup`: perform a cleanup after an already deleted master
* `--force-removal`: force master removal, i.e. ignore topology errors

https://fedorahosted.org/freeipa/ticket/5588
---
 API.txt                  |   4 +-
 VERSION                  |   4 +-
 ipalib/messages.py       |  15 ++
 ipalib/plugins/server.py | 518 ++++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 535 insertions(+), 6 deletions(-)

diff --git a/API.txt b/API.txt
index 5b75413f930d0e9caaffc68023bed8106d786653..ef2945cc7d076c9a4780ff37d6be2ee973554fb7 100644
--- a/API.txt
+++ b/API.txt
@@ -3824,9 +3824,11 @@ output: Output('result', <type 'bool'>, None)
 output: Output('summary', (<type 'unicode'>, <type 'NoneType'>), None)
 output: PrimaryKey('value', None, None)
 command: server_del
-args: 1,2,3
+args: 1,4,3
 arg: Str('cn', attribute=True, cli_name='name', multivalue=True, primary_key=True, query=True, required=True)
+option: Flag('cleanup?', autofill=True, default=False)
 option: Flag('continue', autofill=True, cli_name='continue', default=False)
+option: Flag('force_removal?', autofill=True, default=False)
 option: Str('version?', exclude='webui')
 output: Output('result', <type 'dict'>, None)
 output: Output('summary', (<type 'unicode'>, <type 'NoneType'>), None)
diff --git a/VERSION b/VERSION
index 825aace1b8c78486b37ac1809b664ca18f97523b..bbfa6da80ec8566c7ae754156b2a3863bd3cf75c 100644
--- a/VERSION
+++ b/VERSION
@@ -90,5 +90,5 @@ IPA_DATA_VERSION=20100614120000
 #                                                      #
 ########################################################
 IPA_API_VERSION_MAJOR=2
-IPA_API_VERSION_MINOR=164
-# Last change: simo - add optional string to disable preauth for SPNs
+IPA_API_VERSION_MINOR=165
+# Last change: mbabinsk - extend server-del command
diff --git a/ipalib/messages.py b/ipalib/messages.py
index 5cd0ea1769920c076c62729ed5fe359cd1680723..6014643e5e5800d241a7b52761b08233fb6801b9 100644
--- a/ipalib/messages.py
+++ b/ipalib/messages.py
@@ -357,6 +357,21 @@ class ResultFormattingError(PublicMessage):
     **13019** Unable to correctly format some part of the result
     """
     errno = 13019
+
+
+class MasterRemovalInfo(PublicMessage):
+    """
+    **13020** Informative message printed during removal of IPA master
+    """
+    errno = 13020
+    type = "info"
+
+
+class MasterRemovalWarning(PublicMessage):
+    """
+    **13021** Warning raised during removal of IPA master
+    """
+    errno = 13021
     type = "warning"
 
 
diff --git a/ipalib/plugins/server.py b/ipalib/plugins/server.py
index 93ced8b73049b61fe274c15d84150a892cd34529..22a9c93cbc5a335bf928dab0ce4fde83226bfb75 100644
--- a/ipalib/plugins/server.py
+++ b/ipalib/plugins/server.py
@@ -4,9 +4,11 @@
 
 import dbus
 import dbus.mainloop.glib
+import ldap
+import time
 
-from ipalib import api, crud, errors, messages
-from ipalib import Int, Str
+from ipalib import api, create_api, crud, errors, messages
+from ipalib import Flag, Int, Str
 from ipalib.plugable import Registry
 from ipalib.plugins.baseldap import (
     LDAPSearch,
@@ -16,6 +18,8 @@ from ipalib.plugins.baseldap import (
 from ipalib.request import context
 from ipalib import _, ngettext
 from ipalib import output
+from ipalib.util import create_topology_graph, get_topology_connection_errors
+from ipapython.dn import DN
 
 __doc__ = _("""
 IPA servers
@@ -34,6 +38,87 @@ EXAMPLES:
 register = Registry()
 
 
+def check_hostname_in_masters(hostname, masters):
+    master_cns = {m['cn'][0] for m in masters}
+    return hostname in master_cns
+
+
+def print_connect_errors(topo_errors):
+    msg_lines = []
+    for error in topo_errors:
+        msg_lines.append(
+            "Topology does not allow server %s to replicate with servers:"
+            % error[0]
+        )
+        for srv in error[2]:
+            msg_lines.append("    %s" % srv)
+
+    return "\n".join(msg_lines)
+
+
+def map_masters_to_suffixes(masters):
+    masters_to_suffix = {}
+
+    for master in masters:
+        try:
+            managed_suffixes = master['iparepltopomanagedsuffix_topologysuffix']
+        except KeyError:
+            continue
+
+        for suffix_name in managed_suffixes:
+            try:
+                masters_to_suffix[suffix_name].append(master)
+            except KeyError:
+                masters_to_suffix[suffix_name] = [master]
+
+    return masters_to_suffix
+
+
+def create_topology_graphs(api_instance, masters):
+    """
+    Construct a topology graph for each suffix managed by master
+    :param api_instance: IPA API object
+    :return: dictionary of topology graphs keyed by suffixes managed by
+    the master
+    """
+    suffix_to_masters = map_masters_to_suffixes(masters)
+
+    topology_graphs = {}
+
+    for suffix_name in suffix_to_masters:
+        segments = api_instance.Command.topologysegment_find(
+            suffix_name, sizelimit=0).get('result')
+
+        topology_graphs[suffix_name] = create_topology_graph(
+            suffix_to_masters[suffix_name], segments)
+
+    return topology_graphs
+
+
+def get_topology_errors(api_instance, master_cn, masters):
+    graphs = create_topology_graphs(api_instance, masters)
+
+    topo_errors_by_suffix = {}
+    for suffix_name in graphs:
+        # check topology before removal
+        if master_cn not in graphs[suffix_name].vertices:
+            continue
+
+        orig_errors = get_topology_connection_errors(graphs[suffix_name])
+
+        # after removal
+        try:
+            graphs[suffix_name].remove_vertex(master_cn)
+        except ValueError:
+            pass  # ignore already deleted master, continue to clean
+
+        new_errors = get_topology_connection_errors(graphs[suffix_name])
+
+        topo_errors_by_suffix[suffix_name] = (orig_errors, new_errors)
+
+    return topo_errors_by_suffix
+
+
 @register()
 class server(LDAPObject):
     """
@@ -188,12 +273,439 @@ class server_show(LDAPRetrieve):
         return dn
 
 
+CURR_TOPOLOGY_DISCONNECTED = """
+Replication topology in suffix '{suffix}' is disconnected:
+{errors}"""
+
+REMOVAL_DISCONNECTS_TOPOLOGY = """
+Removal of '{hostname}' leads to disconnected topology in suffix '{suffix}':
+{errors}
+Aborting master removal"""
+
+
 @register()
 class server_del(LDAPDelete):
     __doc__ = _('Delete IPA server.')
-    NO_CLI = True
     msg_summary = _('Deleted IPA server "%(value)s"')
 
+    takes_options = LDAPDelete.takes_options + (
+        Flag(
+            'cleanup?',
+            doc=_('Remove all references to an already deleted master'),
+            default=False,
+        ),
+        Flag(
+            'force_removal?',
+            doc=_('Force master removal'),
+            default=False,
+        ),
+    )
+
+    def _handle_error_in_pre_checks(self, message,
+                                    exc_class=errors.ExecutionError):
+        if self.context.force_removal:
+            self.add_message(
+                messages.MasterRemovalWarning(message=message)
+            )
+        else:
+            raise exc_class(format=_("%(message)s. Master removal aborted."),
+                            message=_(message))
+
+    def _handle_topo_errors(self, master_cn, topo_errors, **options):
+        err_msg = ""
+        for suffix in topo_errors:
+            orig_errors, new_errors = topo_errors[suffix]
+            if orig_errors:
+                err_msg = "\n".join([
+                    err_msg,
+                    CURR_TOPOLOGY_DISCONNECTED.format(
+                        suffix=suffix,
+                        errors=print_connect_errors(orig_errors)
+                    )
+                ])
+            if new_errors:
+                err_msg = "\n".join([
+                    err_msg,
+                    REMOVAL_DISCONNECTS_TOPOLOGY.format(
+                        hostname=master_cn,
+                        suffix=suffix,
+                        errors=print_connect_errors(new_errors)
+                    )
+                ])
+
+        if err_msg:
+            self._handle_error_in_pre_checks(_(err_msg))
+
+    def _ensure_last_services(self, hostname, masters, **options):
+        """
+        1. When deleting master, check if there will be at least one remaining
+           DNS and CA server.
+        2. Pick CA renewal master
+        """
+
+        if not api.env.in_server:
+            return
+
+        from ipaserver.install import cainstance, certs, opendnssecinstance
+        conn = self.obj.backend
+
+        this_services = []
+        other_services = []
+        ca_hostname = None
+
+        for master in masters:
+            master_cn = master['cn'][0]
+            try:
+                services = conn.get_entries(master['dn'], conn.SCOPE_ONELEVEL)
+            except errors.NotFound:
+                continue
+            services_cns = [s.single_value['cn'] for s in services]
+            if master_cn == hostname:
+                this_services = services_cns
+            else:
+                other_services.append(services_cns)
+                if ca_hostname is None and 'CA' in services_cns:
+                    ca_hostname = master_cn
+
+        if 'CA' in this_services and not any(
+                ['CA' in o for o in other_services]):
+            raise errors.ExecutionError(
+                message=_("Deleting this server is not allowed as it would "
+                          "leave your installation without a CA."))
+
+        other_dns = True
+        if 'DNS' in this_services and not any(
+                ['DNS' in o for o in other_services]):
+            other_dns = False
+            self._handle_error_in_pre_checks(
+                _("Deleting this server will leave your installation "
+                  "without a DNS."))
+
+        # test if replica is not DNSSEC master
+        # allow to delete it if is last DNS server
+        if ('DNS' in this_services and other_dns and not
+                self.context.force_removal):
+            dnssec_masters = opendnssecinstance.get_dnssec_key_masters(conn)
+            if hostname in dnssec_masters:
+                self._handle_error_in_pre_checks(
+                    _("Replica is active DNSSEC key master. Uninstall "
+                      "could break your DNS system. Please disable or replace "
+                      "DNSSEC key master first."))
+        ca = cainstance.CAInstance(api.env.realm, certs.NSS_DIR)
+        if ca.is_renewal_master(hostname):
+            try:
+                ca.set_renewal_master(self.api.env.host)
+            except errors.NotFound:
+                ca.set_renewal_master(ca_hostname)
+
+    def pre_callback(self, ldap, dn, *keys, **options):
+        pkey = self.obj.get_primary_key_from_dn(dn)
+
+        self.context.masters = self.api.Command.server_find(
+            u'', sizelimit=0)['result']
+        self.context.force_removal = options.get('force_removal', False)
+
+        if self.context.force_removal:
+            self.add_message(
+                messages.MasterRemovalWarning(
+                    message=_("Forcing removal of {hostname}".format(
+                        hostname=pkey))))
+
+        # gather the topology errors before and after removal
+        self.context.topo_errors = get_topology_errors(self.api, pkey,
+                                                       self.context.masters)
+        self._handle_topo_errors(pkey, self.context.topo_errors, **options)
+
+        # FIXME: check for last services
+        # I would rather wait with this for the Server Roles to limit amount
+        # of server-side code hacking
+        self._ensure_last_services(pkey, self.context.masters, **options)
+
+        return dn
+
+    def _forward_to_other_master(self, *keys, **options):
+        for m in self.context.masters:
+            if m['cn'][0] == self.api.env.host:
+                continue
+
+            remote_api = create_api(mode=None)
+            remote_api.bootstrap(
+                context='server',
+                xmlrpc_uri=u"https://{}/ipa/json".format(m['cn'][0]))
+            remote_api.finalize()
+            rpcclient = remote_api.Backend.rpcclient
+
+            try:
+                rpcclient.connect(delegate=True)
+                return remote_api.Command.server_del(*keys, **options)
+            finally:
+                if rpcclient.is_connected():
+                    rpcclient.disconnect()
+        else:
+            raise errors.ExecutionError(
+                message=_("{master} can not remove itself".format(
+                    master=api.env.host)))
+
+    def execute(self, *keys, **options):
+        self.context.masters = self.api.Command.server_find(
+            u'', sizelimit=0)['result']
+
+        if keys[-1] == self.api.env.host:
+            # the server can not remove itself, try to forward the request
+            # to another master
+            return self._forward_to_other_master(*keys, **options)
+
+        return super(server_del, self).execute(*keys, **options)
+
+    def exc_callback(self, keys, options, exc, call_func, *call_args,
+                     **call_kwargs):
+        if options.get('cleanup', False) and isinstance(exc, errors.NotFound):
+            self.add_message(
+                message=messages.MasterRemovalWarning(
+                    message=_(
+                        "Master already deleted, proceeding with cleanup"
+                    )))
+            return
+
+        raise exc
+
+    def interactive_prompt_callback(self, kw):
+        self.api.Backend.textui.print_plain(
+            _("Removing {server} from replication topology, "
+              "please wait...".format(server=', '.join(kw['cn']))))
+
+    def _check_deleted_segments(self, hostname, masters, topo_errors,
+                                starting_host):
+
+        def wait_for_segment_removal(hostname, master_cns, suffix_name,
+                                     topo_errors):
+            i = 0
+            while True:
+                left = self.api.Command.topologysegment_find(
+                    suffix_name,
+                    iparepltoposegmentleftnode=hostname,
+                    sizelimit=0
+                )['result']
+                right = self.api.Command.topologysegment_find(
+                    suffix_name,
+                    iparepltoposegmentrightnode=hostname,
+                    sizelimit=0
+                )['result']
+
+                # Relax check if topology was or is disconnected. Disconnected
+                # topology can contain segments with already deleted servers
+                # Check only if segments of servers, which can contact this
+                # server, and the deleted server were removed.
+                # This code should handle a case where there was a topology
+                # with a central node(B):  A <-> B <-> C, where A is current
+                # server. After removal of B, topology will be disconnected and
+                # removal of segment B <-> C won't be replicated back to server
+                # A, therefore presence of the segment has to be ignored.
+                if topo_errors[0] or topo_errors[1]:
+                    # use errors after deletion because we don't care if some
+                    # server can't contact the deleted one
+                    cant_contact_me = [e[0] for e in topo_errors[1]
+                                       if starting_host in e[2]]
+                    can_contact_me = set(master_cns) - set(cant_contact_me)
+                    left = [
+                        s for s in left if s['iparepltoposegmentrightnode'][0]
+                        in can_contact_me
+                    ]
+                    right = [
+                        s for s in right if s['iparepltoposegmentleftnode'][0]
+                        in can_contact_me
+                    ]
+
+                if not left and not right:
+                    self.add_message(
+                        messages.MasterRemovalInfo(
+                            message=_("Agreements deleted")
+                        ))
+                    return
+                time.sleep(2)
+                if i == 2:  # taking too long, something is wrong, report
+                    self.add_message(messages.MasterRemovalInfo(
+                        message=_(
+                            "Waiting for removal of replication agreements")))
+                if i > 90:
+                    self.log.info("Taking too long, skipping")
+                    self.log.info("Following segments were not deleted:")
+                    self.add_message(messages.MasterRemovalWarning(
+                        message=_("Following segments were not deleted:")))
+                    for s in left:
+                        self.add_message(messages.MasterRemovalWarning(
+                            message=u"  %s" % s['cn'][0]))
+                    for s in right:
+                        self.add_message(messages.MasterRemovalWarning(
+                            message=u"  %s" % s['cn'][0]))
+                    return
+                i += 1
+
+        suffix_to_masters = map_masters_to_suffixes(masters)
+
+        for suffix_name in suffix_to_masters:
+            suffix_member_cns = [
+                m['cn'][0] for m in suffix_to_masters[suffix_name]
+            ]
+
+            if hostname not in suffix_member_cns:
+                # If the server was already deleted, we can expect that all
+                # removals had been done in previous run and dangling segments
+                # were not deleted.
+                self.log.info(
+                    "Skipping replication agreement deletion check for "
+                    "suffix '{0}'".format(suffix_name))
+                continue
+
+            self.add_message(
+                messages.MasterRemovalInfo(
+                    message=_(
+                        "Checking for deleted segments in suffix '{0}'".format(
+                            suffix_name)
+                    )))
+
+            wait_for_segment_removal(hostname, suffix_member_cns, suffix_name,
+                                     topo_errors[suffix_name])
+
+    def _cleanup_master_ldap_entries(self, conn, master):
+        """
+        This function removes information about the replica in parts
+        of the shared tree that expose it, so clients stop trying to
+        use this replica.
+        """
+
+        # TODO: rewrite log messages to warnings?
+        master_principal = "{}@{}".format(master, self.api.env.realm)
+
+        if master == self.api.env.host:
+            raise errors.ExecutionError(
+                "'{}' can't cleanup self".format(master))
+
+        # delete master kerberos key and all its svc principals
+        try:
+            entries = conn.get_entries(
+                self.api.env.basedn,
+                filter='(krbprincipalname=*/{})'.format(master_principal))
+            if entries:
+                entries.sort(key=lambda x: len(x.dn), reverse=True)
+                for entry in entries:
+                    conn.delete_entry(entry)
+        except errors.NotFound:
+            pass
+        except Exception as e:
+            self.log.error(
+                "Failed to cleanup master principals/keys: {}".format(e)
+            )
+
+        # remove replica memberPrincipal from s4u2proxy configuration
+        s4u2proxy_subtree = DN(self.api.env.container_s4u2proxy,
+                               self.api.env.basedn)
+        dn1 = DN(('cn', 'ipa-http-delegation'), s4u2proxy_subtree)
+        member_principal1 = "HTTP/{}".format(master_principal)
+
+        dn2 = DN(('cn', 'ipa-ldap-delegation-targets'),s4u2proxy_subtree)
+        member_principal2 = "ldap/{}".format(master_principal)
+
+        dn3 = DN(('cn', 'ipa-cifs-delegation-targets'), s4u2proxy_subtree)
+        member_principal3 = "cifs/{}".format(master_principal)
+
+        for (dn, member_principal) in ((dn1, member_principal1),
+                                       (dn2, member_principal2),
+                                       (dn3, member_principal3)):
+            try:
+                mod = [(ldap.MOD_DELETE, 'memberPrincipal', member_principal)]
+                conn.conn.modify_s(str(dn), mod)
+            except (ldap.NO_SUCH_OBJECT, ldap.NO_SUCH_ATTRIBUTE):
+                self.debug(
+                    "Replica (%s) memberPrincipal (%s) not found in %s" %
+                    (master, member_principal, dn))
+            except Exception as e:
+                self.log.error(
+                    "Failed to clean memberPrincipal {} from s4u2proxy entry "
+                    "{}: {}".format(member_principal, dn, e)
+                )
+
+        try:
+            etc_basedn = DN(('cn', 'etc'), self.api.env.basedn)
+            filter = '(dnaHostname=%s)' % master
+            entries = conn.get_entries(
+                etc_basedn, ldap.SCOPE_SUBTREE, filter=filter)
+            if len(entries) != 0:
+                for entry in entries:
+                    conn.delete_entry(entry)
+        except errors.NotFound:
+            pass
+        except Exception as e:
+            self.log.error(
+                "Failed to clean up DNA hostname entries for {}: {}".format(
+                    master, e)
+            )
+
+        try:
+            dn = DN(('cn', 'default'), ('ou', 'profile'), self.api.env.basedn)
+            ret = conn.get_entry(dn)
+            srvlist = ret.single_value.get('defaultServerList', '')
+            srvlist = srvlist[0].split()
+            if master in srvlist:
+                srvlist.remove(master)
+                attr = ' '.join(srvlist)
+                mod = [(ldap.MOD_REPLACE, 'defaultServerList', attr)]
+                conn.conn.modify_s(str(dn), mod)
+        except errors.NotFound:
+            pass
+        except ldap.NO_SUCH_ATTRIBUTE:
+            pass
+        except ldap.TYPE_OR_VALUE_EXISTS:
+            pass
+        except Exception as e:
+            self.log.error(
+                "Failed to remove master {} from server list: {}".format(
+                    master, e)
+            )
+
+    def _cleanup_master_dns_entries(self, hostname, **options):
+        if not self.api.env.in_server or not self.api.Command.dns_is_enabled(
+                **options):
+            return
+
+        realm = self.api.env.realm
+        # TODO: server-side module imports in API code are not nice
+        from ipaserver.install import bindinstance, dnskeysyncinstance
+        try:
+            bind = bindinstance.BindInstance()
+            bind.remove_master_dns_records(hostname, realm, realm.lower())
+            bind.remove_ipa_ca_dns_records(hostname, realm.lower())
+            bind.remove_server_ns_records(hostname)
+
+            keysyncd = dnskeysyncinstance.DNSKeySyncInstance()
+            keysyncd.remove_replica_public_keys(hostname)
+        except Exception as e:
+            import traceback
+            self.log.debug(traceback.format_exc())
+            self.add_message(
+                messages.MasterRemovalWarning(
+                    message=_(
+                        "Failed to cleanup {hostname} DNS entries: "
+                        "{err}".format(hostname=hostname, err=e))))
+            self.add_message(
+                messages.MasterRemovalWarning(
+                    message=_("You may need to manually remove them from the "
+                              "tree")))
+
+    def post_callback(self, ldap, dn, *keys, **options):
+        # remove all leftover references from LDAP
+        self._cleanup_master_ldap_entries(ldap, keys[-1])
+
+        # then check that the replication segments were removed
+        self._check_deleted_segments(
+            keys[-1], self.context.masters, self.context.topo_errors,
+            self.api.env.host)
+
+        # finally try to clean up the leftover DNS entries
+        self._cleanup_master_dns_entries(keys[-1])
+
+        return True
+
 
 @register()
 class server_conncheck(crud.PKQuery):
-- 
2.5.5

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to