On 14/09/16 23:19, Sergio Francisco wrote:
Hi,
We have a deployment of FreeIPA using 3 nodes (Master with more 2 replicas).
Recently, the master node had a problem with the process 'ns-slapd'
consuming 100% of CPU. During this problem, DNS service wasn't working, IPA
admin UI encountered timeout, SSH keys to access the hosts are not being
loaded correctly.
We observed in the logs of "dirsrv" that something related to the cachesize
wasn't enough to the space needed and then ns-slapd started a process to
recover it. We let the server running this operation almost one day and
nothing happened.
Today, we tried to:
1 - remove the failed server from the deployment, using the command below,
but unfortunately, it wasn't possible to do from both the 2 other nodes.
ipa-replica-manage del --force mux-idm-p03.muxi.dc --cacert=/etc/ipa/ca.crt
unexpected error: cannot connect to 'ldaps://localhost.localdomain:636
2 - tried to upgrade the failed server to a most recent version of IPA
using ipa-server-upgrade but it stopped in the step to connect
[5/10]: starting directory server
2016-09-14T13:43:28Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2016-09-14T13:43:28Z DEBUG The ipa-server-upgrade command failed,
exception: error: [Errno 111] Connection refused
2016-09-14T13:43:28Z ERROR [Errno 111] Connection refused
3 - tried to recover the 389-ds database with the command "db_recover -f
-v" but nothing happened.
4 - visited similar threads but none of them helped me
https://www.redhat.com/archives/freeipa-users/2013-May/msg00015.html
https://www.redhat.com/archives/freeipa-users/2015-July/msg00188.html
5 - as we need to urgently recover the service, we tried to rebuild the
failed server, removing and reinstalling all the packages needed by
ipa-server (yum install ipa-server bind bind-dyndb-ldap ipa-server-dns) and
tried to re-join the new server as a replica to receive all the data again,
but it doesn't seems to work.
The other nodes are working well, resolving DNS requests, allowing users to
access the servers using SSH, etc.
Any ideas of what I can do to rebuild the server?
Versions
ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64
ipa-server-dns-4.2.0-15.0.1.el7.centos.19.x86_64
389-ds-base-1.3.4.0-33.el7_2.x86_64
CentOS Linux release 7.2.1511 (Core)
Hi Sergio,
first of all the terms master and replica are misleading. All FreeIPA
servers are masters because the backends (389-ds) are configured to
maintain multi-master replication. The difference between masters may be
in services (CA, DNS, KRA, AD Trust, ...) that was configured on
particular master but the data are synchronized among all masters.
Looking on the steps you've done it would be best to create new master
as a replica of one of the existing masters.
Then you will probably need to enable CRL generating on some master
because this can be enable only on one master and by default is enabled
on first master that is installed with CA. Here you can find more
information and how to:
https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master
HTH,
--
David Kupka
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
