[Freeipa-users] ui timeout issue
I'm seeing an issue with logging into the web UI of ipa. I've been using IPA for 6 months or so in production, and all has been well so far. The last thing i did in terms of IPA was run ipa-dns-install, which completed successfully, but i suspect this issue occured before that i never noticed as it's been a few weeks since i used the UI. I typically check the login page works and ldapsearch works after upgrades, but in this instance the login box is presented, and after entering the credentials it sits doing nothing for a while, then times out with 'internal server error' The only useful log i've managed to find is in /var/log/httpd/error_log [Wed Nov 27 08:41:47 2013] [error] [client (redacted)] Script timed out before returning headers: wsgi.py, referer: https://(redacted)/ipa/ui/ I'm seeing this behaviour on both my master and replica, but they are both identical in terms of package versions and such, so it may not be significant. My system versions: Centos 6.4 x64 ipa-python-3.0.0-26.el6_4.4.x86_64 ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 python-iniparse-0.3.1-2.1.el6.noarch libipa_hbac-1.9.2-82.10.el6_4.x86_64 libipa_hbac-python-1.9.2-82.10.el6_4.x86_64 ipa-client-3.0.0-26.el6_4.4.x86_64 ipa-server-3.0.0-26.el6_4.4.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-admintools-3.0.0-26.el6_4.4.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch bind-dyndb-ldap-2.3-2.el6_4.1.x86_64 bind-9.8.2-0.17.rc1.el6_4.6.x86_64 which are (afaik) all latest for centos 6.4 Oddly, i'm not seeing this behaviour in my virtualbox / vagrant IPA testbed, which has identical version numbers, and wsgi.py in /usr/share/ipa has identical md5sum. Not really sure how to approach debugging this further. Any ideas ? Has anyone else seen this happen ? The ldapsearch, bind dns and everything else seem operational - just the GUI is out of action. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ui timeout issue
On 11/26/2013 03:37 PM, siology.io wrote: I'm seeing an issue with logging into the web UI of ipa. I've been using IPA for 6 months or so in production, and all has been well so far. The last thing i did in terms of IPA was run ipa-dns-install, which completed successfully, but i suspect this issue occured before that i never noticed as it's been a few weeks since i used the UI. I typically check the login page works and ldapsearch works after upgrades, but in this instance the login box is presented, and after entering the credentials it sits doing nothing for a while, then times out with 'internal server error' The only useful log i've managed to find is in /var/log/httpd/error_log [Wed Nov 27 08:41:47 2013] [error] [client (redacted)] Script timed out before returning headers: wsgi.py, referer: https://(redacted)/ipa/ui/ What happens before that in the log? Any DNS lookup or some other lookup? I'm seeing this behaviour on both my master and replica, but they are both identical in terms of package versions and such, so it may not be significant. My system versions: Centos 6.4 x64 ipa-python-3.0.0-26.el6_4.4.x86_64 ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 python-iniparse-0.3.1-2.1.el6.noarch libipa_hbac-1.9.2-82.10.el6_4.x86_64 libipa_hbac-python-1.9.2-82.10.el6_4.x86_64 ipa-client-3.0.0-26.el6_4.4.x86_64 ipa-server-3.0.0-26.el6_4.4.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-admintools-3.0.0-26.el6_4.4.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch bind-dyndb-ldap-2.3-2.el6_4.1.x86_64 bind-9.8.2-0.17.rc1.el6_4.6.x86_64 which are (afaik) all latest for centos 6.4 Oddly, i'm not seeing this behaviour in my virtualbox / vagrant IPA testbed, which has identical version numbers, and wsgi.py in /usr/share/ipa has identical md5sum. Not really sure how to approach debugging this further. Any ideas ? Has anyone else seen this happen ? The ldapsearch, bind dns and everything else seem operational - just the GUI is out of action. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ui timeout issue
On 27 November 2013 10:21, Dmitri Pal d...@redhat.com wrote: On 11/26/2013 03:37 PM, siology.io wrote: I'm seeing an issue with logging into the web UI of ipa. I've been using IPA for 6 months or so in production, and all has been well so far. The last thing i did in terms of IPA was run ipa-dns-install, which completed successfully, but i suspect this issue occured before that i never noticed as it's been a few weeks since i used the UI. I typically check the login page works and ldapsearch works after upgrades, but in this instance the login box is presented, and after entering the credentials it sits doing nothing for a while, then times out with 'internal server error' The only useful log i've managed to find is in /var/log/httpd/error_log [Wed Nov 27 08:41:47 2013] [error] [client (redacted)] Script timed out before returning headers: wsgi.py, referer: https://(redacted)/ipa/ui/ What happens before that in the log? Any DNS lookup or some other lookup? doesn't appear so, no. what makes you suspect that ? I never got as far as doing the ipa-dns-install on the replica. I did it on the master, then went to login and got this issue. It may well be that it (the UI) was broken previously. I couldn't work out how to remove the ipa-dns-install to find out if it magically resumes working though. I'm seeing this behaviour on both my master and replica, but they are both identical in terms of package versions and such, so it may not be significant. My system versions: Centos 6.4 x64 ipa-python-3.0.0-26.el6_4.4.x86_64 ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 python-iniparse-0.3.1-2.1.el6.noarch libipa_hbac-1.9.2-82.10.el6_4.x86_64 libipa_hbac-python-1.9.2-82.10.el6_4.x86_64 ipa-client-3.0.0-26.el6_4.4.x86_64 ipa-server-3.0.0-26.el6_4.4.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-admintools-3.0.0-26.el6_4.4.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch bind-dyndb-ldap-2.3-2.el6_4.1.x86_64 bind-9.8.2-0.17.rc1.el6_4.6.x86_64 which are (afaik) all latest for centos 6.4 Oddly, i'm not seeing this behaviour in my virtualbox / vagrant IPA testbed, which has identical version numbers, and wsgi.py in /usr/share/ipa has identical md5sum. Not really sure how to approach debugging this further. Any ideas ? Has anyone else seen this happen ? The ldapsearch, bind dns and everything else seem operational - just the GUI is out of action. ___ Freeipa-users mailing listFreeipa-users@redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs?www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ui timeout issue
On 11/26/2013 04:32 PM, siology.io wrote: On 27 November 2013 10:21, Dmitri Pal d...@redhat.com mailto:d...@redhat.com wrote: On 11/26/2013 03:37 PM, siology.io http://siology.io wrote: I'm seeing an issue with logging into the web UI of ipa. I've been using IPA for 6 months or so in production, and all has been well so far. The last thing i did in terms of IPA was run ipa-dns-install, which completed successfully, but i suspect this issue occured before that i never noticed as it's been a few weeks since i used the UI. I typically check the login page works and ldapsearch works after upgrades, but in this instance the login box is presented, and after entering the credentials it sits doing nothing for a while, then times out with 'internal server error' The only useful log i've managed to find is in /var/log/httpd/error_log [Wed Nov 27 08:41:47 2013] [error] [client (redacted)] Script timed out before returning headers: wsgi.py, referer: https://(redacted)/ipa/ui/ https://%28redacted%29/ipa/ui/ What happens before that in the log? Any DNS lookup or some other lookup? doesn't appear so, no. what makes you suspect that ? I never got as far as doing the ipa-dns-install on the replica. I did it on the master, then went to login and got this issue. It may well be that it (the UI) was broken previously. I couldn't work out how to remove the ipa-dns-install to find out if it magically resumes working though. A pure speculation: If the UI presents you the form and you fill it then you are definitely talking to the server. When you submit the form the server tries to do kinit on your behalf. It might not be able to determine where its KDC because the DNS configuration is broken in some way and it is now looking at the wrong KDC (may be AD KDC or there is a lack of the server records at all for some reason). I'm seeing this behaviour on both my master and replica, but they are both identical in terms of package versions and such, so it may not be significant. My system versions: Centos 6.4 x64 ipa-python-3.0.0-26.el6_4.4.x86_64 ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 python-iniparse-0.3.1-2.1.el6.noarch libipa_hbac-1.9.2-82.10.el6_4.x86_64 libipa_hbac-python-1.9.2-82.10.el6_4.x86_64 ipa-client-3.0.0-26.el6_4.4.x86_64 ipa-server-3.0.0-26.el6_4.4.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-admintools-3.0.0-26.el6_4.4.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch bind-dyndb-ldap-2.3-2.el6_4.1.x86_64 bind-9.8.2-0.17.rc1.el6_4.6.x86_64 which are (afaik) all latest for centos 6.4 Oddly, i'm not seeing this behaviour in my virtualbox / vagrant IPA testbed, which has identical version numbers, and wsgi.py in /usr/share/ipa has identical md5sum. Not really sure how to approach debugging this further. Any ideas ? Has anyone else seen this happen ? The ldapsearch, bind dns and everything else seem operational - just the GUI is out of action. ___ Freeipa-users mailing list Freeipa-users@redhat.com mailto:Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ http://www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com mailto:Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ui timeout issue
yeah maybe. I do see from the install log of the ipa-dns-install that it changed the /etc/resolv.conf to point to its own ip - which seems a little odd (and unwanted, more importantly). I've changed that back to how it should be and restarted ipa but still nothing. There's no other KDC in the environment that i'm aware of. Certainly, the dns i was using only have the one set of SRV records for ldap and kdc. The bit that puzzles me is how/why that would have affected the replica server also. I asume it's copied the ldap dns data to the replica, but i never installed bind there or bind-dyndb-ldap, or anything else - so i'd expect that to be unaffected but it's also broken now. :-( On 27 November 2013 10:47, Dmitri Pal d...@redhat.com wrote: On 11/26/2013 04:32 PM, siology.io wrote: On 27 November 2013 10:21, Dmitri Pal d...@redhat.com wrote: On 11/26/2013 03:37 PM, siology.io wrote: I'm seeing an issue with logging into the web UI of ipa. I've been using IPA for 6 months or so in production, and all has been well so far. The last thing i did in terms of IPA was run ipa-dns-install, which completed successfully, but i suspect this issue occured before that i never noticed as it's been a few weeks since i used the UI. I typically check the login page works and ldapsearch works after upgrades, but in this instance the login box is presented, and after entering the credentials it sits doing nothing for a while, then times out with 'internal server error' The only useful log i've managed to find is in /var/log/httpd/error_log [Wed Nov 27 08:41:47 2013] [error] [client (redacted)] Script timed out before returning headers: wsgi.py, referer: https://(redacted)/ipa/ui/ What happens before that in the log? Any DNS lookup or some other lookup? doesn't appear so, no. what makes you suspect that ? I never got as far as doing the ipa-dns-install on the replica. I did it on the master, then went to login and got this issue. It may well be that it (the UI) was broken previously. I couldn't work out how to remove the ipa-dns-install to find out if it magically resumes working though. A pure speculation: If the UI presents you the form and you fill it then you are definitely talking to the server. When you submit the form the server tries to do kinit on your behalf. It might not be able to determine where its KDC because the DNS configuration is broken in some way and it is now looking at the wrong KDC (may be AD KDC or there is a lack of the server records at all for some reason). I'm seeing this behaviour on both my master and replica, but they are both identical in terms of package versions and such, so it may not be significant. My system versions: Centos 6.4 x64 ipa-python-3.0.0-26.el6_4.4.x86_64 ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 python-iniparse-0.3.1-2.1.el6.noarch libipa_hbac-1.9.2-82.10.el6_4.x86_64 libipa_hbac-python-1.9.2-82.10.el6_4.x86_64 ipa-client-3.0.0-26.el6_4.4.x86_64 ipa-server-3.0.0-26.el6_4.4.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-admintools-3.0.0-26.el6_4.4.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch bind-dyndb-ldap-2.3-2.el6_4.1.x86_64 bind-9.8.2-0.17.rc1.el6_4.6.x86_64 which are (afaik) all latest for centos 6.4 Oddly, i'm not seeing this behaviour in my virtualbox / vagrant IPA testbed, which has identical version numbers, and wsgi.py in /usr/share/ipa has identical md5sum. Not really sure how to approach debugging this further. Any ideas ? Has anyone else seen this happen ? The ldapsearch, bind dns and everything else seem operational - just the GUI is out of action. ___ Freeipa-users mailing listFreeipa-users@redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs?www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing listFreeipa-users@redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs?www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ui timeout issue
for what it's worth, kinit on the command line of the ipa server works just fine, and detects the realm ok. On 27 November 2013 11:00, siology.io siology...@gmail.com wrote: yeah maybe. I do see from the install log of the ipa-dns-install that it changed the /etc/resolv.conf to point to its own ip - which seems a little odd (and unwanted, more importantly). I've changed that back to how it should be and restarted ipa but still nothing. There's no other KDC in the environment that i'm aware of. Certainly, the dns i was using only have the one set of SRV records for ldap and kdc. The bit that puzzles me is how/why that would have affected the replica server also. I asume it's copied the ldap dns data to the replica, but i never installed bind there or bind-dyndb-ldap, or anything else - so i'd expect that to be unaffected but it's also broken now. :-( On 27 November 2013 10:47, Dmitri Pal d...@redhat.com wrote: On 11/26/2013 04:32 PM, siology.io wrote: On 27 November 2013 10:21, Dmitri Pal d...@redhat.com wrote: On 11/26/2013 03:37 PM, siology.io wrote: I'm seeing an issue with logging into the web UI of ipa. I've been using IPA for 6 months or so in production, and all has been well so far. The last thing i did in terms of IPA was run ipa-dns-install, which completed successfully, but i suspect this issue occured before that i never noticed as it's been a few weeks since i used the UI. I typically check the login page works and ldapsearch works after upgrades, but in this instance the login box is presented, and after entering the credentials it sits doing nothing for a while, then times out with 'internal server error' The only useful log i've managed to find is in /var/log/httpd/error_log [Wed Nov 27 08:41:47 2013] [error] [client (redacted)] Script timed out before returning headers: wsgi.py, referer: https://(redacted)/ipa/ui/ What happens before that in the log? Any DNS lookup or some other lookup? doesn't appear so, no. what makes you suspect that ? I never got as far as doing the ipa-dns-install on the replica. I did it on the master, then went to login and got this issue. It may well be that it (the UI) was broken previously. I couldn't work out how to remove the ipa-dns-install to find out if it magically resumes working though. A pure speculation: If the UI presents you the form and you fill it then you are definitely talking to the server. When you submit the form the server tries to do kinit on your behalf. It might not be able to determine where its KDC because the DNS configuration is broken in some way and it is now looking at the wrong KDC (may be AD KDC or there is a lack of the server records at all for some reason). I'm seeing this behaviour on both my master and replica, but they are both identical in terms of package versions and such, so it may not be significant. My system versions: Centos 6.4 x64 ipa-python-3.0.0-26.el6_4.4.x86_64 ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 python-iniparse-0.3.1-2.1.el6.noarch libipa_hbac-1.9.2-82.10.el6_4.x86_64 libipa_hbac-python-1.9.2-82.10.el6_4.x86_64 ipa-client-3.0.0-26.el6_4.4.x86_64 ipa-server-3.0.0-26.el6_4.4.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-admintools-3.0.0-26.el6_4.4.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch bind-dyndb-ldap-2.3-2.el6_4.1.x86_64 bind-9.8.2-0.17.rc1.el6_4.6.x86_64 which are (afaik) all latest for centos 6.4 Oddly, i'm not seeing this behaviour in my virtualbox / vagrant IPA testbed, which has identical version numbers, and wsgi.py in /usr/share/ipa has identical md5sum. Not really sure how to approach debugging this further. Any ideas ? Has anyone else seen this happen ? The ldapsearch, bind dns and everything else seem operational - just the GUI is out of action. ___ Freeipa-users mailing listFreeipa-users@redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs?www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing listFreeipa-users@redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs?www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com
Re: [Freeipa-users] ui timeout issue
On 11/26/2013 05:15 PM, siology.io wrote: for what it's worth, kinit on the command line of the ipa server works just fine, and detects the realm ok. OK then let us rule out DNS for a moment. Have you checked the KDC log to see whether the authentication actually occurred? If kinit works, I suspect it works too but worth checking. May be there are some problems with memcached after the form based authentication to cache the authentication. KDC log would show whether the kinit and follow up service ticket request for LDAP access actually occurred. Also what about SELinux any suspicious AVC? On 27 November 2013 11:00, siology.io http://siology.io siology...@gmail.com mailto:siology...@gmail.com wrote: yeah maybe. I do see from the install log of the ipa-dns-install that it changed the /etc/resolv.conf to point to its own ip - which seems a little odd (and unwanted, more importantly). I've changed that back to how it should be and restarted ipa but still nothing. There's no other KDC in the environment that i'm aware of. Certainly, the dns i was using only have the one set of SRV records for ldap and kdc. The bit that puzzles me is how/why that would have affected the replica server also. I asume it's copied the ldap dns data to the replica, but i never installed bind there or bind-dyndb-ldap, or anything else - so i'd expect that to be unaffected but it's also broken now. :-( On 27 November 2013 10:47, Dmitri Pal d...@redhat.com mailto:d...@redhat.com wrote: On 11/26/2013 04:32 PM, siology.io http://siology.io wrote: On 27 November 2013 10:21, Dmitri Pal d...@redhat.com mailto:d...@redhat.com wrote: On 11/26/2013 03:37 PM, siology.io http://siology.io wrote: I'm seeing an issue with logging into the web UI of ipa. I've been using IPA for 6 months or so in production, and all has been well so far. The last thing i did in terms of IPA was run ipa-dns-install, which completed successfully, but i suspect this issue occured before that i never noticed as it's been a few weeks since i used the UI. I typically check the login page works and ldapsearch works after upgrades, but in this instance the login box is presented, and after entering the credentials it sits doing nothing for a while, then times out with 'internal server error' The only useful log i've managed to find is in /var/log/httpd/error_log [Wed Nov 27 08:41:47 2013] [error] [client (redacted)] Script timed out before returning headers: wsgi.py, referer: https://(redacted)/ipa/ui/ https://%28redacted%29/ipa/ui/ What happens before that in the log? Any DNS lookup or some other lookup? doesn't appear so, no. what makes you suspect that ? I never got as far as doing the ipa-dns-install on the replica. I did it on the master, then went to login and got this issue. It may well be that it (the UI) was broken previously. I couldn't work out how to remove the ipa-dns-install to find out if it magically resumes working though. A pure speculation: If the UI presents you the form and you fill it then you are definitely talking to the server. When you submit the form the server tries to do kinit on your behalf. It might not be able to determine where its KDC because the DNS configuration is broken in some way and it is now looking at the wrong KDC (may be AD KDC or there is a lack of the server records at all for some reason). I'm seeing this behaviour on both my master and replica, but they are both identical in terms of package versions and such, so it may not be significant. My system versions: Centos 6.4 x64 ipa-python-3.0.0-26.el6_4.4.x86_64 ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 python-iniparse-0.3.1-2.1.el6.noarch libipa_hbac-1.9.2-82.10.el6_4.x86_64 libipa_hbac-python-1.9.2-82.10.el6_4.x86_64 ipa-client-3.0.0-26.el6_4.4.x86_64 ipa-server-3.0.0-26.el6_4.4.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-admintools-3.0.0-26.el6_4.4.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch bind-dyndb-ldap-2.3-2.el6_4.1.x86_64 bind-9.8.2-0.17.rc1.el6_4.6.x86_64 which are (afaik) all latest for centos 6.4 Oddly, i'm not seeing this behaviour in my virtualbox / vagrant IPA testbed, which has identical version numbers, and wsgi.py in
