Dear community, we just moved our infrastructure (about 200 node cluster plus about 30 workstations) from NIS to FreeIPA (version 4.1.4 on FC 21).
We have two IPA servers (called "ipa" and "ipa2" both paravirtualized on Xen4). Approx once a day, the Kerberos service on the primary server suddenly stops working and I am unable to re-start the service. Only a "full" reboot helps and during that, the Kerberos shutdown takes about 2 minutes (unsure if it really finishes or if it's the final timeout of the shutdown script). Trying to collect as many log messages as possible: Jul 22 10:52:06 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): AS_REQ (4 etypes {18 17 16 23}) 132.195.124.213: LOOKING_UP_CLIENT: host/proton.pleiades.uni-wuppertal...@pleiades.uni-wuppertal.de for krbtgt/pleiades.uni-wuppertal...@pleiades.uni-wuppertal.de, Server error Jul 22 10:52:11 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): AS_REQ (4 etypes {18 17 16 23}) 132.195.125.171: LOOKING_UP_CLIENT: host/wn161.pleiades.uni-wuppertal...@pleiades.uni-wuppertal.de for krbtgt/pleiades.uni-wuppertal...@pleiades.uni-wuppertal.de, Server error [root@ipa ~]# systemctl status krb5kdc.service ● krb5kdc.service - Kerberos 5 KDC Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled) Active: failed (Result: exit-code) since Mi 2015-07-22 10:54:22 CEST; 10s ago Process: 11910 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=1/FAILURE) Main PID: 1114 (code=exited, status=0/SUCCESS) Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de krb5kdc[11910]: krb5kdc: cannot initialize realm PLEIADES.UNI-WUPPERTAL.DE - see log file for details Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: krb5kdc.service: control process exited, code=exited status=1 Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: Failed to start Kerberos 5 KDC. Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: Unit krb5kdc.service entered failed state. Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: krb5kdc.service failed. [root@ipa ~]# tail -f /var/log/krb5kdc.log Jul 22 10:52:11 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): closing down fd 13 Jul 22 10:52:11 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): DISPATCH: repeated (retransmitted?) request from 132.195.124.213, resending previous response Jul 22 10:52:11 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): DISPATCH: repeated (retransmitted?) request from 132.195.125.171, resending previous response Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](debug): Got signal to request exit Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): closing down fd 10 Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): closing down fd 11 Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): closing down fd 9 Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): closing down fd 8 Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de krb5kdc[1114](info): shutting down krb5kdc: Server error - while fetching master key K/M for realm PLEIADES.UNI-WUPPERTAL.DE [root@ipa ~]# journalctl -xe Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen and drop on 1 v6wildcard :: UDP 123 Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen normally on 2 lo 127.0.0.1 UDP 123 Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen normally on 3 eth0 132.195.124.12 UDP 123 Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen normally on 4 lo ::1 UDP 123 Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listen normally on 5 eth0 fe80::216:3eff:fe14:c27a UDP 123 Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: Listening on routing socket on fd #22 for interface updates Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de systemd[1]: Started Network Time Service. -- Subject: Unit ntpd.service has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit ntpd.service has finished starting up. -- -- The start-up result is done. Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: 0.0.0.0 c016 06 restart Jul 22 10:53:15 ipa.pleiades.uni-wuppertal.de ntpd[11903]: 0.0.0.0 c012 02 freq_set ntpd -23.557 PPM Jul 22 10:53:16 ipa.pleiades.uni-wuppertal.de ntpd[11903]: 0.0.0.0 c615 05 clock_sync Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de systemd[1]: Stopping Kerberos 5 KDC... -- Subject: Unit krb5kdc.service has begun shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit krb5kdc.service has begun shutting down. Jul 22 10:54:12 ipa.pleiades.uni-wuppertal.de systemd[1]: Starting Kerberos 5 KDC... -- Subject: Unit krb5kdc.service has begun with start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit krb5kdc.service has begun starting up. Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de krb5kdc[11910]: krb5kdc: cannot initialize realm PLEIADES.UNI-WUPPERTAL.DE - see log file for details Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: krb5kdc.service: control process exited, code=exited status=1 Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: Failed to start Kerberos 5 KDC. -- Subject: Unit krb5kdc.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit krb5kdc.service has failed. -- -- The result is failed. Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: Unit krb5kdc.service entered failed state. Jul 22 10:54:22 ipa.pleiades.uni-wuppertal.de systemd[1]: krb5kdc.service failed. [root@ipa ~]# [root@ipa ~]# rpm -qi freeipa-server Name : freeipa-server Version : 4.1.4 Release : 1.fc21 Architecture: x86_64 Install Date: Di 28 Apr 2015 14:30:33 CEST Group : System Environment/Base Size : 4521059 License : GPLv3+ Signature : RSA/SHA256, Do 26 Mär 2015 23:58:02 CET, Key ID 89ad4e8795a43f54 Source RPM : freeipa-4.1.4-1.fc21.src.rpm Build Date : Do 26 Mär 2015 16:16:19 CET Build Host : buildhw-07.phx2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : http://www.freeipa.org/ Summary : The IPA authentication server Description : IPA is an integrated solution to provide centrally managed Identity (machine, user, virtual machines, groups, authentication credentials), Policy (configuration settings, access control information) and Audit (events, logs, analysis thereof). If you are installing an IPA server you need to install this package (in other words, most people should NOT install this package). [root@ipa ~]# We already enlarged the capacity of the primary server (now two exclusive CPU cores and 8 GB RAM). Any idea is appreciated, we are pretty new to IPA. Kind regards, Torsten -- <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> <> <> <> Dr. Torsten Harenberg harenb...@physik.uni-wuppertal.de <> <> Bergische Universitaet <> <> FB C - Physik Tel.: +49 (0)202 439-3521 <> <> Gaussstr. 20 Fax : +49 (0)202 439-2811 <> <> 42097 Wuppertal <> <> <> <><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><> -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project