[Freeipa-users] Get Creation Time / Last Login Time for Users
Hello, We're looking for a way to get last login time and creation time for users configured in FreeIPA. This information doesn't seem to be in the WebUI and ipa user-status only provides limited information (last failed/successful logins in seconds since epoch). Is there a supported way to get this information? Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Servers intermittently losing connection to IPA
Sumit, We found a resolution for this and I'm dropping it here for posterity. After some digging, it turns out that our ipa server and ipa replica were returning different IPs for systems in the environment in DNS requests (one returned internal results, one returned external results). After resolving this our intermittent connectivity issue went away. So it seems that in some cases, the incorrect IP was being returned for LDAP requests. One additional item found here, it seems that the timeout to resolve an address (from the sssd logs) is 6 seconds. Can this be raised? Thanks, Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> On Thu, Apr 21, 2016 at 7:47 AM, Sumit Bose <sb...@redhat.com> wrote: > On Wed, Apr 20, 2016 at 02:18:28PM -0400, Jeff Hallyburton wrote: > > Sumit, > > > > Raised the debug level to 10 and let it run for about 24 hours. > Uploading > > the last 2000~ lines of the sssd_domain.com.log. Thanks for your help! > > Can you send the related krb5_child log file as well? > > bye, > Sumit > > > > > https://pastebin.com/MD6N1Dj7 > > > > Jeff Hallyburton > > Strategic Systems Engineer > > Bloomip Inc. > > Web: http://www.bloomip.com > > > > Engineering Support: supp...@bloomip.com > > Billing Support: bill...@bloomip.com > > Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/ > > > > > > On Tue, Apr 19, 2016 at 1:14 PM, Jeff Hallyburton < > > jeff.hallybur...@bloomip.com> wrote: > > > > > Sumit, > > > > > > Raised the debug level to 10 and let it run for about 24 hours. > Uploading > > > the full sssd_domain.com.log. Thanks for your help! > > > > > > Jeff > > > > > > Jeff Hallyburton > > > Strategic Systems Engineer > > > Bloomip Inc. > > > Web: http://www.bloomip.com > > > > > > Engineering Support: supp...@bloomip.com > > > Billing Support: bill...@bloomip.com > > > Customer Support Portal: https://my.bloomip.com < > http://my.bloomip.com/> > > > > > > On Mon, Apr 18, 2016 at 10:58 AM, Sumit Bose <sb...@redhat.com> wrote: > > > > > >> On Fri, Apr 15, 2016 at 04:47:42PM -0400, Jeff Hallyburton wrote: > > >> > After setting debug_level=8, this is what I see in the > sssd_domain_log: > > >> > > >> Unfortunately the domain log and the krb5_child log do not relate to > > >> each other. > > >> > > >> > > > >> > (Fri Apr 15 20:10:46 2016) [sssd[be[example.com]]] > > >> [child_handler_setup] > > >> > (0x2000): Setting up signal handler up for pid [32382] > > >> > > > >> > > >> > > >> > > >> > > > >> > (Fri Apr 15 20:32:47 2016) [[sssd[krb5_child[32731 > [k5c_setup_fast] > > >> > (0x0100): SSSD_KRB5_FAST_PRINCIPAL is set to [host/ > > >> > jump02.west-2.production.example@example.com] > > >> > > > >> > > >> ... > > >> > > >> > (Fri Apr 15 20:32:47 2016) [[sssd[krb5_child[32731 > > >> [get_and_save_tgt] > > >> > (0x0400): krb5_get_init_creds_password returned [-1765328324} during > > >> > pre-auth. > > >> > > > >> > > > >> > Can you shed any light on this? > > >> > > > >> > > >> In the domain log the child with the pid 32382 is started to run a > > >> pre-authentication request. The request is needed to find out which > kind > > >> of authentication types are available for the user, e.g. password or > > >> 2-factor authentication with the OTP token. The request in the child > > >> with the PID 32731 looks like a real authentication request with > returns > > >> with an error code -1765328324 which just means 'Generic error' but > > >> might have cause SSSD to go offline. > > >> > > >> I would like to ask you to run the test again with debug_level=10 in > the > > >> [domain/...] section of sssd.conf which would enable some low level > > >> Kerberos tracing messages which might help to understand what kind of > > >> 'Generic error' was hit here. Additionally I would like ask you to > send > > >> the full log files as attachment or in an archive which would hep be > to > > >> better navigate through them. > > >> > > >> bye, > > >> Sumit > > >> > > > > > > > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Servers intermittently losing connection to IPA
Sumit, Raised the debug level to 10 and let it run for about 24 hours. Uploading the last 2000~ lines of the sssd_domain.com.log. Thanks for your help! https://pastebin.com/MD6N1Dj7 Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> On Tue, Apr 19, 2016 at 1:14 PM, Jeff Hallyburton < jeff.hallybur...@bloomip.com> wrote: > Sumit, > > Raised the debug level to 10 and let it run for about 24 hours. Uploading > the full sssd_domain.com.log. Thanks for your help! > > Jeff > > Jeff Hallyburton > Strategic Systems Engineer > Bloomip Inc. > Web: http://www.bloomip.com > > Engineering Support: supp...@bloomip.com > Billing Support: bill...@bloomip.com > Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> > > On Mon, Apr 18, 2016 at 10:58 AM, Sumit Bose <sb...@redhat.com> wrote: > >> On Fri, Apr 15, 2016 at 04:47:42PM -0400, Jeff Hallyburton wrote: >> > After setting debug_level=8, this is what I see in the sssd_domain_log: >> >> Unfortunately the domain log and the krb5_child log do not relate to >> each other. >> >> > >> > (Fri Apr 15 20:10:46 2016) [sssd[be[example.com]]] >> [child_handler_setup] >> > (0x2000): Setting up signal handler up for pid [32382] >> > >> >> >> >> > >> > (Fri Apr 15 20:32:47 2016) [[sssd[krb5_child[32731 [k5c_setup_fast] >> > (0x0100): SSSD_KRB5_FAST_PRINCIPAL is set to [host/ >> > jump02.west-2.production.example@example.com] >> > >> >> ... >> >> > (Fri Apr 15 20:32:47 2016) [[sssd[krb5_child[32731 >> [get_and_save_tgt] >> > (0x0400): krb5_get_init_creds_password returned [-1765328324} during >> > pre-auth. >> > >> > >> > Can you shed any light on this? >> > >> >> In the domain log the child with the pid 32382 is started to run a >> pre-authentication request. The request is needed to find out which kind >> of authentication types are available for the user, e.g. password or >> 2-factor authentication with the OTP token. The request in the child >> with the PID 32731 looks like a real authentication request with returns >> with an error code -1765328324 which just means 'Generic error' but >> might have cause SSSD to go offline. >> >> I would like to ask you to run the test again with debug_level=10 in the >> [domain/...] section of sssd.conf which would enable some low level >> Kerberos tracing messages which might help to understand what kind of >> 'Generic error' was hit here. Additionally I would like ask you to send >> the full log files as attachment or in an archive which would hep be to >> better navigate through them. >> >> bye, >> Sumit >> > > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD Crash Causing Inaccessibility
Understood. Unfortunately, this event has been diagnosed and mitigated, so re-occurance is unlikely. Will respond to this thread if we see any repeats however, totally understand the need for further information here. Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> On Fri, Jan 29, 2016 at 5:44 PM, Jakub Hrozek <jhro...@redhat.com> wrote: > On Fri, Jan 29, 2016 at 01:49:15PM +0100, Lukas Slebodnik wrote: > > On (28/01/16 18:37), Jeff Hallyburton wrote: > > >Application logs showed this to be due to an OOM error, so no need to > chase > > >this further. Thanks for the quick response! > > > > > Even though it was OOM. > > I would still be interested in version of sssd. > > "access after free error" is bed error. > > > > Do you have a coredump. It might be stored > > by abrt or systemd-coredumpd (coredumpctl) > > This problem reminds me of: > https://fedorahosted.org/sssd/ticket/2886 > > Sadly, that one was also a one-time condition and we could never get to > the root cause from the corefile. > > I agree with Lukas the core would be nice to see.. > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD Crash Causing Inaccessibility
Lukas, Installed versions of sssd: # rpm -qa | grep -i sssd sssd-common-1.13.0-40.el7_2.1.x86_64 sssd-ipa-1.13.0-40.el7_2.1.x86_64 sssd-1.13.0-40.el7_2.1.x86_64 sssd-krb5-common-1.13.0-40.el7_2.1.x86_64 sssd-ad-1.13.0-40.el7_2.1.x86_64 sssd-ldap-1.13.0-40.el7_2.1.x86_64 sssd-proxy-1.13.0-40.el7_2.1.x86_64 python-sssdconfig-1.13.0-40.el7_2.1.noarch sssd-client-1.13.0-40.el7_2.1.x86_64 sssd-common-pac-1.13.0-40.el7_2.1.x86_64 sssd-krb5-1.13.0-40.el7_2.1.x86_64 No core dumps unfortunately. Thanks, Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> On Fri, Jan 29, 2016 at 7:49 AM, Lukas Slebodnik <lsleb...@redhat.com> wrote: > On (28/01/16 18:37), Jeff Hallyburton wrote: > >Application logs showed this to be due to an OOM error, so no need to > chase > >this further. Thanks for the quick response! > > > Even though it was OOM. > I would still be interested in version of sssd. > "access after free error" is bed error. > > Do you have a coredump. It might be stored > by abrt or systemd-coredumpd (coredumpctl) > > LS > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
[Freeipa-users] IPA Web Portal using outdated ciphers, breaking with some clients
Hi, We're also seeing that the free-ipa web-portal is using TLS 1.2 by default, which is being flagged as insecure / obsolete. This also seems to be causing some clients (some instances of Chrome) to fail logins: [Fri Jan 29 18:34:26.638350 2016] [:error] [pid 6603] SSL Library Error: -12286 No common encryption algorithm(s) with client What do we need to do to update this to TLS 1.3? Thanks, Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] IPA Web Portal using outdated ciphers, breaking with some clients
Rob, Chrome is flagging this, and given the error (I've attached a copy) its probably due to the cipher suite (possibly specifically that it uses SHA1). This article has more details and is consistent with what we're seeing: http://security.stackexchange.com/questions/83831/google-chrome-your-connection-to-website-is-encrypted-with-obsolete-cryptograph We've also seen similar issues come up with other applications during penetration scans (e.g., Qualys) which is why I've noted it here. Thanks, Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> On Fri, Jan 29, 2016 at 2:36 PM, Rob Crittenden <rcrit...@redhat.com> wrote: > Jeff Hallyburton wrote: > > Hi, > > > > We're also seeing that the free-ipa web-portal is using TLS 1.2 by > > default, which is being flagged as insecure / obsolete. This also seems > > to be causing some clients (some instances of Chrome) to fail logins: > > > > [Fri Jan 29 18:34:26.638350 2016] [:error] [pid 6603] SSL Library Error: > > -12286 No common encryption algorithm(s) with client > > > > > > What do we need to do to update this to TLS 1.3? > > TLS 1.2 insecure/obsolete? Flagged by what? Need more info on what the > handshake looks like and what the server configuration is. > > AFAIK 1.3 is still in draft form. > > rob > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
[Freeipa-users] SSSD Crash Causing Inaccessibility
We saw the following happen on a system today, and wanted to follow up: System became unresponsive to ssh logins with the error: ssh -v incentives01 OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 4: Applying options for * debug1: Connecting to incentives01 [172.31.9.16] port 22. debug1: Connection established. debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1 debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x0400 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5-...@openssh.com none debug1: kex: client->server aes128-ctr hmac-md5-...@openssh.com none debug1: kex: curve25519-sha...@libssh.org need=16 dh_need=16 debug1: kex: curve25519-sha...@libssh.org need=16 dh_need=16 debug1: sending SSH2_MSG_KEX_ECDH_INIT debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ECDSA 89:e0:f8:25:21:db:c9:46:67:14:38:0c:c1:f4:f7:51 debug1: Host 'incentives01' is known and matches the ECDSA host key. debug1: Found key in /home/jeff.hallyburton/.ssh/known_hosts:7 debug1: ssh_ecdsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received This is a private computer system which is restricted to authorized individuals. Actual or attempted unauthorized use of this computer system will result in criminal and/or civil prosecution. We reserve the right to view, monitor and record activity on the system without notice or permission. Any information obtained by monitoring, reviewing or recording is subject to review by law enforcement organizations in connection with the investigation or prosecution of possible criminal activity on this system. If you are not an authorized user of this system or do not consent to continued monitoring, disconnect at this time. debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic Received disconnect from 172.31.9.16: 2: Too many authentication failures for jeff.hallyburton Ultimately we rebooted the node to restore connectivity. After we were back in, we're seeing that sssd crashed due what looks like a memory allocation error: /var/log/sssd/sssd.log # cat /var/log/sssd/sssd.log (Thu Jan 28 20:15:56 2016) [sssd] [mt_svc_sigkill] (0x0010): [enervee.com][620] is not responding to SIGTERM. Sending SIGKILL. (Thu Jan 28 20:16:27 2016) [sssd] [talloc_log_fn] (0x0010): talloc: access after free error - first free may be at src/monitor/monitor.c:2760 (Thu Jan 28 20:16:27 2016) [sssd] [talloc_log_fn] (0x0010): Bad talloc magic value - access after free /var/log/secure Jan 28 20:05:48 incentives01 sshd[26145]: Timeout, client not responding. Jan 28 20:05:48 incentives01 sshd[26142]: pam_unix(sshd:session): session closed for user Jan 28 20:16:28 incentives01 sshd[14504]: Timeout, client not responding. Jan 28 20:16:28 incentives01 sshd[14501]: pam_systemd(sshd:session): Failed to release session: Connection timed out Jan 28 20:16:28 incentives01 sshd[14501]: pam_unix(sshd:session): session closed for user Jan 28 20:16:28 incentives01 sshd[14501]: pam_sss(sshd:session): Request to sssd failed. Bad address Jan 28 20:16:29 incentives01 sshd[14501]: fatal: login_init_entry: Cannot find user Jan 28 20:21:40 incentives01 sshd[26882]: Invalid user from 172.31.8.34 The system may have simply run out of ram, but wanted to check to see if there were any known or contributing issues. Thanks, Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> -- Manage your
Re: [Freeipa-users] SSSD Crash Causing Inaccessibility
Application logs showed this to be due to an OOM error, so no need to chase this further. Thanks for the quick response! Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> On Thu, Jan 28, 2016 at 6:22 PM, Lukas Slebodnik <lsleb...@redhat.com> wrote: > On (28/01/16 16:25), Jeff Hallyburton wrote: > >We saw the following happen on a system today, and wanted to follow up: > > > >System became unresponsive to ssh logins with the error: > > > >ssh -v incentives01 > > > //snip > > ># cat /var/log/sssd/sssd.log > > > >(Thu Jan 28 20:15:56 2016) [sssd] [mt_svc_sigkill] (0x0010): [enervee.com > ][620] > >is not responding to SIGTERM. Sending SIGKILL. > > > >(Thu Jan 28 20:16:27 2016) [sssd] [talloc_log_fn] (0x0010): talloc: access > >after free error - first free may be at src/monitor/monitor.c:2760 > > > > > >(Thu Jan 28 20:16:27 2016) [sssd] [talloc_log_fn] (0x0010): Bad talloc > >magic value - access after free > > > There was a crash in sssd. It might explain why you cannot login. > Which version of sssd do you have? > > LS > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Free-IPA failover succeeds, but ssh is broken?
Janelle, The proxy suggestion was spot on. After that things seem to work normally. Thanks! Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> On Sun, Jan 17, 2016 at 9:58 AM, Janelle <janellenicol...@gmail.com> wrote: > Hi, > > Try commenting out the proxy command in /etc/ssh/ssh_config > > The sssd proxy of ssh is buggy as can be. > > ~J > > > On Jan 17, 2016, at 05:24, Jakub Hrozek <jhro...@redhat.com> wrote: > > > > > >> On 16 Jan 2016, at 02:21, Jeff Hallyburton < > jeff.hallybur...@bloomip.com> wrote: > >> > >> Having finished setting up an ipa server and replica, we're trying to > test failover to ensure that HA works as expected. We've been able to > verify the replication agreements and auto-discovery are working, and both > servers are picked up as expected at install time. > >> > >> That said, we're seeing some oddities with failover. Once I shut down > the ipa service on the main ipa server, I get most requests completing > after about a 2 min window. I am able to: > >> > >> 1. Authenticate to our jump server and get a kerberos ticket > >> 2. kinit successfully as other users > >> > >> However, whenever I try to ssh to another system within our domain, ssh > breaks with the following error: > >> > >> $ ssh -vvv automation01 > >> OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 > >> debug1: Reading configuration data /etc/ssh/ssh_config > >> debug1: /etc/ssh/ssh_config line 5: Applying options for * > >> debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy > -p 22 automation01 > >> debug1: permanently_drop_suid: 158701 > >> debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1 > >> debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1 > >> debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1 > >> debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1 > >> debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1 > >> debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1 > >> debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1 > >> debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type > -1 > >> debug1: Enabling compatibility mode for protocol 2.0 > >> debug1: Local version string SSH-2.0-OpenSSH_6.6.1 > >> ssh_exchange_identification: Connection closed by remote host > > > > Did you crank up debug level on the machine where sshd is running and > see if anything is logged then? > > > >> > >> Nothing is logged in either /var/log/messages or /var/log/secure when > this happens, so I'm unsure where to begin debugging. Can you offer any > insight? > >> > >> Thanks, > >> > >> Jeff > >> > >> Jeff Hallyburton > >> Strategic Systems Engineer > >> Bloomip Inc. > >> Web: http://www.bloomip.com > >> > >> Engineering Support: supp...@bloomip.com > >> Billing Support: bill...@bloomip.com > >> Customer Support Portal: https://my.bloomip.com > >> -- > >> Manage your subscription for the Freeipa-users mailing list: > >> https://www.redhat.com/mailman/listinfo/freeipa-users > >> Go to http://freeipa.org for more info on the project > > > > > > -- > > Manage your subscription for the Freeipa-users mailing list: > > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
[Freeipa-users] Free-IPA failover succeeds, but ssh is broken?
Having finished setting up an ipa server and replica, we're trying to test failover to ensure that HA works as expected. We've been able to verify the replication agreements and auto-discovery are working, and both servers are picked up as expected at install time. That said, we're seeing some oddities with failover. Once I shut down the ipa service on the main ipa server, I get most requests completing after about a 2 min window. I am able to: 1. Authenticate to our jump server and get a kerberos ticket 2. kinit successfully as other users However, whenever I try to ssh to another system within our domain, ssh breaks with the following error: $ ssh -vvv automation01 OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 5: Applying options for * debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 automation01 debug1: permanently_drop_suid: 158701 debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1 debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1 ssh_exchange_identification: Connection closed by remote host Nothing is logged in either /var/log/messages or /var/log/secure when this happens, so I'm unsure where to begin debugging. Can you offer any insight? Thanks, Jeff Jeff Hallyburton Strategic Systems Engineer Bloomip Inc. Web: http://www.bloomip.com Engineering Support: supp...@bloomip.com Billing Support: bill...@bloomip.com Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/> -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] FreeIPA Replica / HA Issues
Petr,
Thanks for the info. This is in fact probably what's happening in our
case. That said, is there any supported way of manually setting up
failover at this time? Is it hard, or simply impossible?
Thanks,
Jeff
Jeff Hallyburton
Strategic Systems Engineer
Bloomip Inc.
Web: http://www.bloomip.com
Engineering Support: supp...@bloomip.com
Billing Support: bill...@bloomip.com
Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/>
On Thu, Jan 14, 2016 at 2:06 AM, Petr Spacek <pspa...@redhat.com> wrote:
> Hello,
>
>
> this log is weird:
>
> On 14.1.2016 03:02, Jeff Hallyburton wrote:
> >> 2016-01-14T00:45:35Z DEBUG [IPA Discovery]
> >> 2016-01-14T00:45:35Z DEBUG Starting IPA discovery with domain=
> west-2.production.example.com, servers=None, hostname=
> test.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG Search for LDAP SRV record in
> west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG Search DNS for SRV record of _ldap._
> tcp.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: 0 100 389
> ipa1.west-2.production.example.com.
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: 10 100 389
> ipa2.west-2.production.example.com.
> >> 2016-01-14T00:45:35Z DEBUG [Kerberos realm search]
> >> 2016-01-14T00:45:35Z DEBUG Search DNS for TXT record of _
> kerberos.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: "EXAMPLE.COM"
> >> 2016-01-14T00:45:35Z DEBUG Search DNS for SRV record of _kerberos._
> udp.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: 10 100 88
> ipa2.west-2.production.example.com.
> >> 2016-01-14T00:45:35Z DEBUG DNS record found: 0 100 88
> ipa1.west-2.production.example.com.
> >> 2016-01-14T00:45:35Z DEBUG [LDAP server check]
> >> 2016-01-14T00:45:35Z DEBUG Verifying that
> ipa1.west-2.production.example.com (realm EXAMPLE.COM) is an IPA server
> >> 2016-01-14T00:45:35Z DEBUG Init LDAP connection to:
> ipa1.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG Search LDAP server for IPA base DN
> >> 2016-01-14T00:45:35Z DEBUG Check if naming context 'dc=example,dc=com'
> is for IPA
> >> 2016-01-14T00:45:35Z DEBUG Naming context 'dc=example,dc=com' is a
> valid IPA context
> >> 2016-01-14T00:45:35Z DEBUG Search for (objectClass=krbRealmContainer)
> in dc=example,dc=com (sub)
> >> 2016-01-14T00:45:35Z DEBUG Found: cn=EXAMPLE.COM
> ,cn=kerberos,dc=example,dc=com
> >> 2016-01-14T00:45:35Z DEBUG Discovery result: Success; server=
> ipa1.west-2.production.example.com, domain=west-2.production.example.com,
> kdc=ipa2.west-2.production.example.com,ipa1.west-2.production.example.com,
> basedn=dc=example,dc=com
> >> 2016-01-14T00:45:35Z DEBUG Validated servers:
> ipa1.west-2.production.example.com
> >> 2016-01-14T00:45:35Z DEBUG will use discovered domain:
> west-2.production.example.com
>
> It looks that your IPA domain & realm is "example.com" and "EXAMPLE.COM",
> is
> that correct?
>
> Looking further ...
>
> > 2016-01-14T00:45:39Z DEBUG Writing Kerberos configuration to
> /etc/krb5.conf:
> > 2016-01-14T00:45:39Z DEBUG #File modified by ipa-client-install
> >
> > includedir /var/lib/sss/pubconf/krb5.include.d/
> >
> > [libdefaults]
> > default_realm = EXAMPLE.COM
> > dns_lookup_realm = true
> > dns_lookup_kdc = true
> > rdns = false
> > ticket_lifetime = 24h
> > forwardable = yes
> > udp_preference_limit = 0
> > default_ccache_name = KEYRING:persistent:%{uid}
> >
> >
> > [realms]
> > EXAMPLE.COM = {
> > pkinit_anchors = FILE:/etc/ipa/ca.crt
> >
> > }
> >
> >
> > [domain_realm]
> > .west-2.production.example.com = EXAMPLE.COM
> > west-2.production.example.com = EXAMPLE.COM
>
> Hmm, this is going to be wild guess, but let's try it:
> Do you have DNS SRV records in domain west-2.production.example.com but
> not in
> DNS domain example.com?
>
> That would probably cause this kind of problem.
>
> Generally it is necessary to put _kerberos TXT + SRV records into the
> (primary) DNS domain specified during IPA installation. Then use --domain
> option during ipa-client-install.
>
> --server is generally discouraged as it disables DNS SRV lookup and makes
> failover hard or impossible.
>
> --domain is just a hint for the installer where to start looking for DNS
> SRV
> records and allows full automatic failover.
>
>
> The autodiscovery is quite messy and needs to be imporoved in next
> versions.
> https://fedorahosted.org/freeipa/ticket/5270 should avoid the need to
> specify
> --domain when Kerberos TXT record is in DNS ... Stay tuned :-)
>
> --
> Petr^2 Spacek
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
[Freeipa-users] FreeIPA Replica / HA Issues
We've deployed a FreeIPA server in a client infrastructure and now we're
working on making that setup HA. We've created a replica and I can verify
that the replica has connectivity to the existing master and ensured that
the auto-discovery DNS records are set up for LDAP / Kerberos / etc, but
I'm having a couple of issues with clients:
1. ipa-client-install fails with the following error whenever a server is
not explicitly specified (though explicitly specifying either the original
master OR the replica works fine):
trying https://ipa1.west-2.production.example.com/ipa/json
Cannot connect to the server due to Kerberos error: Kerberos error:
Kerberos error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM"',
-1765328230)/. Trying with delegate=True
trying https://ipa1.west-2.production.example.com/ipa/json
Second connect with delegate=True also failed: Kerberos error: Kerberos
error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM"',
-1765328230)/
Cannot connect to the IPA server RPC interface: Kerberos error: Kerberos
error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM"',
-1765328230)/
Installation failed. Rolling back changes.
Failed to list certificates in /etc/ipa/nssdb: Command ''/usr/bin/certutil'
'-d' '/etc/ipa/nssdb' '-L'' returned non-zero exit status 255
Unenrolling client from IPA server
Unenrolling host failed: Error obtaining initial credentials: Cannot find
KDC for requested realm.
What we see in the install logs is:
2016-01-14T00:45:39Z INFO Configured /etc/krb5.conf for IPA realm
EXAMPLE.COM
2016-01-14T00:45:39Z DEBUG Starting external process
2016-01-14T00:45:39Z DEBUG args='keyctl' 'search' '@s' 'user'
'ipa_session_cookie:host/test.west-2.production.example@example.com'
2016-01-14T00:45:39Z DEBUG Process finished, return code=1
2016-01-14T00:45:39Z DEBUG stdout=
2016-01-14T00:45:39Z DEBUG stderr=keyctl_search: Required key not available
2016-01-14T00:45:39Z DEBUG Starting external process
2016-01-14T00:45:39Z DEBUG args='/usr/bin/certutil' '-d' '/tmp/tmpCJNEzU'
'-N' '-f' '/tmp/tmpPN7H8R'
2016-01-14T00:45:39Z DEBUG Process finished, return code=0
2016-01-14T00:45:39Z DEBUG stdout=
2016-01-14T00:45:39Z DEBUG stderr=
2016-01-14T00:45:39Z DEBUG Starting external process
2016-01-14T00:45:39Z DEBUG args='/usr/bin/certutil' '-d' '/tmp/tmpCJNEzU'
'-A' '-n' 'CA certificate 1' '-t' 'C,,'
2016-01-14T00:45:39Z DEBUG Process finished, return code=0
2016-01-14T00:45:39Z DEBUG stdout=
2016-01-14T00:45:39Z DEBUG stderr=
2016-01-14T00:45:39Z DEBUG Starting external process
2016-01-14T00:45:39Z DEBUG args='keyctl' 'search' '@s' 'user'
'ipa_session_cookie:host/test.west-2.production.example@example.com'
2016-01-14T00:45:39Z DEBUG Process finished, return code=1
2016-01-14T00:45:39Z DEBUG stdout=
2016-01-14T00:45:39Z DEBUG stderr=keyctl_search: Required key not available
2016-01-14T00:45:39Z DEBUG failed to find session_cookie in persistent
storage for principal 'host/test.west-2.production.example@example.com'
2016-01-14T00:45:39Z INFO trying
https://ipa1.west-2.production.example.com/ipa/json
2016-01-14T00:45:39Z INFO Cannot connect to the server due to Kerberos
error: Kerberos error: Kerberos error: ('Unspecified GSS failure. Minor
code may provide more information', 851968)/('Cannot find KDC for realm "
EXAMPLE.COM"', -1765328230)/. Trying with delegate=True
2016-01-14T00:45:39Z INFO trying
https://ipa1.west-2.production.example.com/ipa/json
2016-01-14T00:45:39Z WARNING Second connect with delegate=True also failed:
Kerberos error: Kerberos error: ('Unspecified GSS failure. Minor code may
provide more information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM"',
-1765328230)/
2016-01-14T00:45:39Z ERROR Cannot connect to the IPA server RPC interface:
Kerberos error: Kerberos error: ('Unspecified GSS failure. Minor code may
provide more information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM"',
-1765328230)/
2016-01-14T00:45:39Z ERROR Installation failed. Rolling back changes.
2016-01-14T00:45:39Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2016-01-14T00:45:39Z DEBUG Starting external process
2016-01-14T00:45:39Z DEBUG args='ipa-client-automount' '--uninstall'
'--debug'
2016-01-14T00:45:40Z DEBUG Process finished, return code=0
2016-01-14T00:45:40Z DEBUG stdout=Restoring configuration
2. Related to this, all of our existing clients have been configured with
explicit server= statements, meaning that they don't pick up the replica
either. Is there any way to manually fix this post installation, or will
we simply have to uninstall and reinstall the ipa client?
Thanks,
Jeff
Jeff Hallyburton
Strategic Systems Engineer
Bloomip
Re: [Freeipa-users] FreeIPA Replica / HA Issues
Rob,
Full log is attached.
Jeff
Jeff Hallyburton
Strategic Systems Engineer
Bloomip Inc.
Web: http://www.bloomip.com
Engineering Support: supp...@bloomip.com
Billing Support: bill...@bloomip.com
Customer Support Portal: https://my.bloomip.com <http://my.bloomip.com/>
On Wed, Jan 13, 2016 at 8:35 PM, Rob Crittenden <rcrit...@redhat.com> wrote:
> Jeff Hallyburton wrote:
> > We've deployed a FreeIPA server in a client infrastructure and now we're
> > working on making that setup HA. We've created a replica and I can
> > verify that the replica has connectivity to the existing master and
> > ensured that the auto-discovery DNS records are set up for LDAP /
> > Kerberos / etc, but I'm having a couple of issues with clients:
> >
> > 1. ipa-client-install fails with the following error whenever a server
> > is not explicitly specified (though explicitly specifying either the
> > original master OR the replica works fine):
> >
> > trying https://ipa1.west-2.production.example.com/ipa/json
> >
> > Cannot connect to the server due to Kerberos error: Kerberos error:
> > Kerberos error: ('Unspecified GSS failure. Minor code may provide more
> > information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> > <http://EXAMPLE.COM>"', -1765328230)/. Trying with delegate=True
> >
> > trying https://ipa1.west-2.production.example.com/ipa/json
> >
> > Second connect with delegate=True also failed: Kerberos error: Kerberos
> > error: ('Unspecified GSS failure. Minor code may provide more
> > information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> > <http://EXAMPLE.COM>"', -1765328230)/
> >
> > Cannot connect to the IPA server RPC interface: Kerberos error: Kerberos
> > error: ('Unspecified GSS failure. Minor code may provide more
> > information', 851968)/('Cannot find KDC for realm "EXAMPLE.COM
> > <http://EXAMPLE.COM>"', -1765328230)/
> >
> > Installation failed. Rolling back changes.
> >
> > Failed to list certificates in /etc/ipa/nssdb: Command
> > ''/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-L'' returned non-zero exit
> > status 255
> >
> > Unenrolling client from IPA server
> >
> > Unenrolling host failed: Error obtaining initial credentials: Cannot
> > find KDC for requested realm.
> >
> >
> > What we see in the install logs is:
> >
> > 2016-01-14T00:45:39Z INFO Configured /etc/krb5.conf for IPA realm
> > EXAMPLE.COM <http://EXAMPLE.COM>
> >
> > 2016-01-14T00:45:39Z DEBUG Starting external process
> >
> > 2016-01-14T00:45:39Z DEBUG args='keyctl' 'search' '@s' 'user'
> > 'ipa_session_cookie:host/test.west-2.production.example@example.com
> > <mailto:test.west-2.production.example@example.com>'
> >
> > 2016-01-14T00:45:39Z DEBUG Process finished, return code=1
> >
> > 2016-01-14T00:45:39Z DEBUG stdout=
> >
> > 2016-01-14T00:45:39Z DEBUG stderr=keyctl_search: Required key not
> available
> >
> >
> > 2016-01-14T00:45:39Z DEBUG Starting external process
> >
> > 2016-01-14T00:45:39Z DEBUG args='/usr/bin/certutil' '-d'
> > '/tmp/tmpCJNEzU' '-N' '-f' '/tmp/tmpPN7H8R'
> >
> > 2016-01-14T00:45:39Z DEBUG Process finished, return code=0
> >
> > 2016-01-14T00:45:39Z DEBUG stdout=
> >
> > 2016-01-14T00:45:39Z DEBUG stderr=
> >
> > 2016-01-14T00:45:39Z DEBUG Starting external process
> >
> > 2016-01-14T00:45:39Z DEBUG args='/usr/bin/certutil' '-d'
> > '/tmp/tmpCJNEzU' '-A' '-n' 'CA certificate 1' '-t' 'C,,'
> >
> > 2016-01-14T00:45:39Z DEBUG Process finished, return code=0
> >
> > 2016-01-14T00:45:39Z DEBUG stdout=
> >
> > 2016-01-14T00:45:39Z DEBUG stderr=
> >
> > 2016-01-14T00:45:39Z DEBUG Starting external process
> >
> > 2016-01-14T00:45:39Z DEBUG args='keyctl' 'search' '@s' 'user'
> > 'ipa_session_cookie:host/test.west-2.production.example@example.com
> > <mailto:test.west-2.production.example@example.com>'
> >
> > 2016-01-14T00:45:39Z DEBUG Process finished, return code=1
> >
> > 2016-01-14T00:45:39Z DEBUG stdout=
> >
> > 2016-01-14T00:45:39Z DEBUG stderr=keyctl_search: Required key not
> available
> >
> >
> > 2016-01-14T00:45:39Z DEBUG failed to find session_cookie in persistent
> > storage for principal
> > 'host/test.west-2.production.example@example.com
> > <mailto:test.west-2.production.example@example.com>'
> >
> > 2016-01-14T00:45:39Z INFO trying
> > https://ipa1.west-2.production.ex
