Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
On Sun, Feb 12, 2012 at 2:15 PM, Marco Pizzoli marco.pizz...@gmail.comwrote: Hi Alexander, On Sat, Feb 11, 2012 at 11:54 PM, Alexander Bokovoy aboko...@redhat.comwrote: On Sat, 11 Feb 2012, Marco Pizzoli wrote: Hi, Today I booted my FreeIPA 2.1.4 system on Fedora16 and now I'm failing in having it started. [root@freeipa01 ~]# systemctl | grep ipa ipa.service loaded failed failedIdentity, Policy, Audit /var/log/messages [cut] Feb 11 12:15:13 freeipa01 systemd[1]: PID file /run/sendmail.pid not readable (yet?) after start. Feb 11 12:15:13 freeipa01 ntpd_intres[821]: host name not found: 0.fedora.pool.ntp.org Feb 11 12:15:13 freeipa01 ntpd_intres[821]: host name not found: 1.fedora.pool.ntp.org Feb 11 12:15:13 freeipa01 ntpd_intres[821]: host name not found: 2.fedora.pool.ntp.org Feb 11 12:15:14 freeipa01 systemd[1]: PID file /run/sm-client.pid not readable (yet?) after start. Feb 11 12:15:29 freeipa01 ipactl[998]: Failed to read data from Directory Service: Unknown error when retrieving list of services from LDAP: [Errno 111] Connection refused Feb 11 12:15:29 freeipa01 ipactl[998]: Shutting down Feb 11 12:15:29 freeipa01 ipactl[998]: Starting Directory Service Feb 11 12:15:29 freeipa01 systemd[1]: ipa.service: main process exited, code=exited, status=1 Feb 11 12:15:29 freeipa01 systemd[1]: Unit ipa.service entered failed state. Feb 11 12:15:29 freeipa01 systemd[1]: Startup finished in 2s 327ms 887us (kernel) + 4s 398ms 198us (initrd) + 40s 949ms 673us (userspace) = 47s 675ms 758us. [cut] /var/log/dirsrv/slapd-my_user_dir/errors [cut] [11/Feb/2012:12:15:27 +0100] - 389-Directory/1.2.10.a6 B2011.353.1631 starting up [11/Feb/2012:12:15:27 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. /var/log/dirsrv/slapd-my_pki_dir/errors [cut] [11/Feb/2012:12:15:27 +0100] - 389-Directory/1.2.10.a6 B2011.353.1631 starting up [11/Feb/2012:12:15:27 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. dmesg output [cut] [ 17.440200] systemd-tmpfiles[743]: Successfully loaded SELinux database in 14ms 981us, size on heap is 485K. [ 17.593118] systemd-tmpfiles[743]: Two or more conflicting lines for /var/run/dirsrv configured, ignoring. [ 17.593225] systemd-tmpfiles[743]: Two or more conflicting lines for /var/lock/dirsrv configured, ignoring. [cut] Any help? Did you try 'ipactl start' afterwards? Yes, same as before. I'm not sure what has caused 389-ds database issue but from the log excerpts it looks like 389-ds was able to fix those. Fedora 16 stable updates got freeipa 2.1.4-5 and 389-ds 1.2.10-rc1 tonight. Now, I did a full upgrade of the system but I'm encountering quite the same problem. The interesting thing is that the 389-ds upgrade produced a log full of interesting info about what the problem is. Please find my log here: http://pastebin.com/ueH87Q05 I'm running a system with less than 1GB RAM [root@freeipa01 ~]# free -m total used free sharedbuffers cached Mem: 869758110 0 42561 -/+ buffers/cache:154714 Swap: 2015 0 2015 I'm curious to know if is an opportunity to recover the system. If no, I have no problems in erase and recreate. Thanks again Marco I'm having the same issue with another freeipa setup which was installed directly from the updates-testing repository. He was working correctly once installed but then, after the first power-on after the installation, no working from the 389-ds side. [12/Feb/2012:16:19:44 +0100] - 389-Directory/1.2.10.rc1 B2012.035.328 starting up [12/Feb/2012:16:19:44 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. [12/Feb/2012:16:19:44 +0100] - libdb: unable to join the environment ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
On Sun, 12 Feb 2012, Marco Pizzoli wrote: I'm having the same issue with another freeipa setup which was installed directly from the updates-testing repository. He was working correctly once installed but then, after the first power-on after the installation, no working from the 389-ds side. [12/Feb/2012:16:19:44 +0100] - 389-Directory/1.2.10.rc1 B2012.035.328 starting up [12/Feb/2012:16:19:44 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. [12/Feb/2012:16:19:44 +0100] - libdb: unable to join the environment So there is something fishy with 389-ds shutdown on reboots? Am I correct in assuming that you had FreeIPA working after install, then power cycled the VM and after restart it didn't come back online? Was there anything specific about shutdown? Anything similar to https://fedorahosted.org/freeipa/ticket/2302 ? -- / Alexander Bokovoy ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
On Sun, Feb 12, 2012 at 5:41 PM, Alexander Bokovoy aboko...@redhat.comwrote: On Sun, 12 Feb 2012, Marco Pizzoli wrote: I'm having the same issue with another freeipa setup which was installed directly from the updates-testing repository. He was working correctly once installed but then, after the first power-on after the installation, no working from the 389-ds side. [12/Feb/2012:16:19:44 +0100] - 389-Directory/1.2.10.rc1 B2012.035.328 starting up [12/Feb/2012:16:19:44 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. [12/Feb/2012:16:19:44 +0100] - libdb: unable to join the environment So there is something fishy with 389-ds shutdown on reboots? Am I correct in assuming that you had FreeIPA working after install, then power cycled the VM and after restart it didn't come back online? Well, just to be clear, each time I talked about reboot actually I intended shutdown -h now and powering on the day after. Was there anything specific about shutdown? Anything similar to https://fedorahosted.org/freeipa/ticket/2302 ? I don't get hangs or other type of similar evidences. My system just complete (correctly, it seems) a shutdown sequence. I am not yet an expert about systemd, so I don't know if it's just going to kill the service if it doesn't respond in a specific time to a request to shut down. I'm working with more than one virtual machine active on my not-so-new laptop, so the promptness of response is very low... If you want me to do any kind of test, just let me know. Thanks Marco ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
On Sun, 12 Feb 2012, Marco Pizzoli wrote: On Sun, Feb 12, 2012 at 5:41 PM, Alexander Bokovoy aboko...@redhat.comwrote: On Sun, 12 Feb 2012, Marco Pizzoli wrote: I'm having the same issue with another freeipa setup which was installed directly from the updates-testing repository. He was working correctly once installed but then, after the first power-on after the installation, no working from the 389-ds side. [12/Feb/2012:16:19:44 +0100] - 389-Directory/1.2.10.rc1 B2012.035.328 starting up [12/Feb/2012:16:19:44 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. [12/Feb/2012:16:19:44 +0100] - libdb: unable to join the environment So there is something fishy with 389-ds shutdown on reboots? Am I correct in assuming that you had FreeIPA working after install, then power cycled the VM and after restart it didn't come back online? Well, just to be clear, each time I talked about reboot actually I intended shutdown -h now and powering on the day after. Was there anything specific about shutdown? Anything similar to https://fedorahosted.org/freeipa/ticket/2302 ? I don't get hangs or other type of similar evidences. My system just complete (correctly, it seems) a shutdown sequence. I am not yet an expert about systemd, so I don't know if it's just going to kill the service if it doesn't respond in a specific time to a request to shut down. I'm working with more than one virtual machine active on my not-so-new laptop, so the promptness of response is very low... If you want me to do any kind of test, just let me know. If you could reproduce similar results with new VM, it would be good to get access to the 389-ds database in question and exact steps to reproduce the failure. -- / Alexander Bokovoy ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
On Sun, Feb 12, 2012 at 6:00 PM, Alexander Bokovoy aboko...@redhat.comwrote: On Sun, 12 Feb 2012, Marco Pizzoli wrote: On Sun, Feb 12, 2012 at 5:41 PM, Alexander Bokovoy aboko...@redhat.com wrote: On Sun, 12 Feb 2012, Marco Pizzoli wrote: I'm having the same issue with another freeipa setup which was installed directly from the updates-testing repository. He was working correctly once installed but then, after the first power-on after the installation, no working from the 389-ds side. [12/Feb/2012:16:19:44 +0100] - 389-Directory/1.2.10.rc1 B2012.035.328 starting up [12/Feb/2012:16:19:44 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. [12/Feb/2012:16:19:44 +0100] - libdb: unable to join the environment So there is something fishy with 389-ds shutdown on reboots? Am I correct in assuming that you had FreeIPA working after install, then power cycled the VM and after restart it didn't come back online? Well, just to be clear, each time I talked about reboot actually I intended shutdown -h now and powering on the day after. Was there anything specific about shutdown? Anything similar to https://fedorahosted.org/freeipa/ticket/2302 ? I don't get hangs or other type of similar evidences. My system just complete (correctly, it seems) a shutdown sequence. I am not yet an expert about systemd, so I don't know if it's just going to kill the service if it doesn't respond in a specific time to a request to shut down. I'm working with more than one virtual machine active on my not-so-new laptop, so the promptness of response is very low... If you want me to do any kind of test, just let me know. If you could reproduce similar results with new VM, it would be good to get access to the 389-ds database in question and exact steps to reproduce the failure. I can start the VM setup right now, but please explain more in detail what I do need to do for this trial. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
On Sun, 12 Feb 2012, Marco Pizzoli wrote: I don't get hangs or other type of similar evidences. My system just complete (correctly, it seems) a shutdown sequence. I am not yet an expert about systemd, so I don't know if it's just going to kill the service if it doesn't respond in a specific time to a request to shut down. I'm working with more than one virtual machine active on my not-so-new laptop, so the promptness of response is very low... If you want me to do any kind of test, just let me know. If you could reproduce similar results with new VM, it would be good to get access to the 389-ds database in question and exact steps to reproduce the failure. I can start the VM setup right now, but please explain more in detail what I do need to do for this trial. Ideally, install Fedora 16 and apply all updates. Then connect over ssh with something like this: $ ssh root@freeipa-test-vm | tee -a ~/freeipa-test-vm-session.log and perform FreeIPA packages install, ipa-server-install, and all operations that caused the data corruption. You can logout and enter over ssh multiple times, every time using the command above to ensure that log is appended. This log will show what has happened on the console as you performed install and configuration. In addition to it /var/log will contain number of files (ipaserver-*.log, ipaclient-*.log, pki*.log, pki-ca/*, dirsrv/*, etc) with logs relevant to FreeIPA operations. Then /etc/dirsrv/ would contain 389-ds instances' data stores. Thanks in advance. -- / Alexander Bokovoy ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
Here they are. I think that it is not worth sending an attachment of over 1.2MB to the entire list, even if I don't have any personal data in them. Thanks. Could you please edit /usr/sbin/ipactl and change timeout parameter at lines 125 and 128 to something greater than 6? Maybe 10 or even 15... The parameter is seconds to time out: .. wait_for_open_socket(lurl.hostport, timeout=6) .. wait_for_open_ports(host, [int(port)], timeout=6) .. Looks like your VM is so slow that ipactl simply times out to wait for the directory server to respond. We've seen this before with some other VMs. Good catch! I tried with 25, but same result :-( I tried with 45 and now it is up! Please, could you confirm that the following exited is not bad thing: [root@freeipa04 ~]# systemctl|grep ipa ipa.service loaded active *exited*Identity, Policy, Audit ipa_kpasswd.service loaded active running IPA Kerberos password service Thanks a lot! Marco -- _ Non รจ forte chi non cade, ma chi cadendo ha la forza di rialzarsi. Jim Morrison ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
On Sun, 12 Feb 2012, Marco Pizzoli wrote: Here they are. I think that it is not worth sending an attachment of over 1.2MB to the entire list, even if I don't have any personal data in them. Thanks. Could you please edit /usr/sbin/ipactl and change timeout parameter at lines 125 and 128 to something greater than 6? Maybe 10 or even 15... The parameter is seconds to time out: .. wait_for_open_socket(lurl.hostport, timeout=6) .. wait_for_open_ports(host, [int(port)], timeout=6) .. Looks like your VM is so slow that ipactl simply times out to wait for the directory server to respond. We've seen this before with some other VMs. Good catch! I tried with 25, but same result :-( I tried with 45 and now it is up! Please, could you confirm that the following exited is not bad thing: [root@freeipa04 ~]# systemctl|grep ipa ipa.service loaded active *exited*Identity, Policy, Audit ipa_kpasswd.service loaded active running IPA Kerberos password service *exited* is fine, it is /usr/sbin/ipactl exited after running the startup sequence. Would you mind to file a ticket against FreeIPA to make this time out configurable in /etc/ipa/default.conf? This is something that we can't predict in all cases so this would be per-system setting. -- / Alexander Bokovoy ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA not starting - probably 389ds cause
On Sun, Feb 12, 2012 at 10:26 PM, Alexander Bokovoy aboko...@redhat.comwrote: On Sun, 12 Feb 2012, Marco Pizzoli wrote: Here they are. I think that it is not worth sending an attachment of over 1.2MB to the entire list, even if I don't have any personal data in them. Thanks. Could you please edit /usr/sbin/ipactl and change timeout parameter at lines 125 and 128 to something greater than 6? Maybe 10 or even 15... The parameter is seconds to time out: .. wait_for_open_socket(lurl.hostport, timeout=6) .. wait_for_open_ports(host, [int(port)], timeout=6) .. Looks like your VM is so slow that ipactl simply times out to wait for the directory server to respond. We've seen this before with some other VMs. Good catch! I tried with 25, but same result :-( I tried with 45 and now it is up! Please, could you confirm that the following exited is not bad thing: [root@freeipa04 ~]# systemctl|grep ipa ipa.service loaded active *exited*Identity, Policy, Audit ipa_kpasswd.service loaded active running IPA Kerberos password service *exited* is fine, it is /usr/sbin/ipactl exited after running the startup sequence. Ok, thanks. Would you mind to file a ticket against FreeIPA to make this time out configurable in /etc/ipa/default.conf? This is something that we can't predict in all cases so this would be per-system setting. Done. https://fedorahosted.org/freeipa/ticket/2375 For the record, in creating a new ticket I notice that I can specify as affected version only versions 2.0 and alpha3. Marco ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
