Hi Dmitri, Prasun Thanks for those tickets. I have commented Dimitri's with a reference to this thread.
Cheers Chris From: Dmitri Pal <d...@redhat.com> To: freeipa-users@redhat.com Date: 07.06.2015 22:33 Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Solved Sent by: freeipa-users-boun...@redhat.com On 06/05/2015 01:47 PM, Prasun Gera wrote: I had faced a similar issue a month ago, for which I had created a ticket. https://fedorahosted.org/freeipa/ticket/4956 On Fri, Jun 5, 2015 at 7:30 AM, Alexander Bokovoy < aboko...@redhat.com> wrote: On Fri, 05 Jun 2015, Christopher Lamb wrote: Hi Martin Thanks for updating the documenation! The suggested solution works not only my test servers, but also "in the real world". This morning I migrated the last production server (ipa host) to the new FreeIPA KDC. Just out of idle curiosity, why is the rm -f /var/lib/sss/db/* step required on our EL 7.1 + ipa-client 4.1 boxes, but not on our older EL 6.5 + ipa-client 3.3.3 machines? Is the problem down to sssd? (on the EL 6.5 machines we are running sssd 1.9.2, while on EL 7.1 we have sssd 1.12.2 I think there are more object types supported by newer SSSD versions which aren't invalidated like users or groups. Cheers Chris From: Martin Kosek <mko...@redhat.com> To: Christopher Lamb/Switzerland/IBM@IBMCH, Rob Crittenden <rcrit...@redhat.com>, freeipa-users@redhat.com Cc: Jakub Hrozek <jhro...@redhat.com> Date: 05.06.2015 08:06 Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Solved On 06/04/2015 07:34 PM, Christopher Lamb wrote: Hi All I can now report back success (at least on my throwaway EL7.1 test VM). To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC to a new FreeIPA 4.1 KDC 3 steps are required: 1) ipa-client-install --uninstall 2) rm -f /var/lib/sss/db/* 3) ipa-client-install --server ldap.my.example.com --domain my.example.com -N Having done this, my free-ipa user successfully authenticates (e.g. ssh remote login with free-ipa user / password To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required. Kudos and thanks go to Rob C for suggesting step 2. (Note that the directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as suggested earlier in this thread. Cool! Thanks for reaching back. I added this advice to the FreeIPA Troubleshooting guide too: http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client Cheers Chris From: Martin Kosek <mko...@redhat.com> To: Christopher Lamb/Switzerland/IBM@IBMCH, freeipa-users@redhat.com Cc: Jakub Hrozek <jhro...@redhat.com>, Rob Crittenden <rcrit...@redhat.com> Date: 03.06.2015 10:39 Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Not Solved On 06/03/2015 10:30 AM, Christopher Lamb wrote: Hi all This is a quick(ish) note to bring everybody up to speed on this issue. Yesterday we had some private mail exchange on this issue as I did not wish to broadcast the krb5 and ipa install logs to the user list. The basic situation is that we are in the process of migrating from an FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As discussed in a thread some weeks ago we did not do this by replicating (as perhaps we should have done). Instead we migrated the users across. We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined to the old KDC. We are now in the process of migrating these hosts to the new 4.1 KDC. Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these joining to the new KDC was trouble free, taking a few minutes each. After joining the new KDC FreeIPA users authenticated properly. We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that were joined direct to the new 4.1 KDC, never having been joined of the 3.3.3 KDC. These were also trouble free. The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1 hosts that were originally joined to the 3.3.3 KDC, and must be moved to join the 4.1 KDC. These machines no longer authenticate valid FreeIPA users. I have been able to reproduce this behaviour with a freshly setup VM joined first to the 3.3.3 KDC, then moved to the 4.1 KDC. While the errors show in the krb5 child logs indicate that the password is incorrect, the same user / password is happily accepted by all the other hosts. It seems that in the process of moving / migrating the EL 7.1 / ipa-client 4.1 from the old KDC to the new KDC, "something" is left behind that causes problems. We have seen indications in the install logs that the kinit steps called during ipa-client install are getting responses from the wrong (old) KDC, and not from the new KDC. Frustratingly. over the weekend i managed to get one of the problem EL 7.1 boxes to work. However I can't work out exactly what I was that I did that did the trick. However it seems that some kind of major de-install / cleanup + reinstall of the ipa-client may be needed. Rob has suggested that as part of such a cleanup I should do "rm -f /var/lib/sssd/db/*". I will test this later today and report back. Thanks to Rob, Jakub, Martin, Alexander et al for their help and suggestions so far. Chris Thanks for the background. The pain you are getting is exactly the reason why migration via replication to RHEL-7.1 is a better choice :-) Please let us know the result, I am curious how this works out. From: Martin Kosek < mko...@redhat.com> To: Christopher Lamb/Switzerland/IBM@IBMCH, freeipa-users@redhat.com, Jakub Hrozek < jhro...@redhat.com> Date: 03.06.2015 09:34 Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Not Solved On 06/02/2015 06:15 PM, Christopher Lamb wrote: Hi Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause of this problem. Let's call them HOST09 and HOST10 Both are mimimum installs of EL7.1, with NTPD installed and configured. HOST09 had ipa-client 4.1 installed via yum, and was configured to use our new FreeIPA 4.1 server, right from the start. --> My FreeIPA user authenticates successfully against this machine. HOST10 had ipa-client 4.1 installed as a dependency of one of our standard config packages, and was first set to use our old FreeIPA 3.3.3 server. --> My FreeIPA user authenticates successfully. against this machine. I then de-registered HOST10 from the FreeIPA 3.1 server, and registered against the new FreeIPA 4.1 server --> My FreeIPA users does NOT authenticate successfully. This replicates well the behaviour I saw with my production servers, namely a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new 4.1 FreeIPA server authenticate properly. b) EL 7.1 hosts with ipa-client 4.1 first registered against the old 3.3.3 FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT authenticate properly Chris Hello, This is really strange. What I do not fully understand is what is the "registration against a FreeIPA server". What server you install IPA client should matter if the deployment is set up properly. The host enrollment entry should simply replicate to whole infrastructure. The only thing that will probably differ is sssd.conf and krb5.conf as they will have different primary server set up, based on what your DNS setup is. It rather seems that the "reregistration" is what causes the issue. It looks like something cleanup problem during the process. I will let Jakub to help here, I would suggest including the SSSD logs from the failed login, it may help. ----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52 ----- From: Christopher Lamb/Switzerland/IBM@IBMCH To: Jakub Hrozek <jhro...@redhat.com> Cc: freeipa-users@redhat.com Date: 02.06.2015 10:40 Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Not Solved Sent by: freeipa-users-boun...@redhat.com Hi Jakub Yes root login works, that's how I've been getting into the box. Surprisingly, kinit with my user seems to work on that box. After entering my password when prompted, it returns to the commandline without error. However if I try kinit with another FreeIPA user, then instead of prompting for a password, it gives "Generic preauthentication failure while getting initial credentials" error. Having set debug_level=10, when I try and ssh in with my FreeIPA user, I find errors like "Retrieving host .... with result: .. Matching credential not found" "Received error from KDC ... Additional pre-authentication required" "Received error from KDC... Decrypt integrity check failed" "Received error code 1432158219" Cheers Chris From: Jakub Hrozek <jhro...@redhat.com> To: Christopher Lamb/Switzerland/IBM@IBMCH Cc: freeipa-users@redhat.com Date: 02.06.2015 09:50 Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Not Solved On Tue, Jun 02, 2015 at 09:43:48AM +0200, Christopher Lamb wrote: Hi Jakub The same user / password works with all our FreeIPA hosts - just this one box is the problem. So the password should be good. Of course a type is always possible (especially for strong passwords), but I have tried many times which should eliminate the odd password typo. The user / password should also be good for both the old and the new FreeIPA Server. Interesting, can you add debug_level=10 to the domain section of sssd.conf? Then krb5_child.log should show Kerberos tracing info including which exact KDC SSSD was talking to. As I can neither log in direct, or via ssh to this box with my FreeIPA user, I assume Kinit with my user won't work- i will try later in the day. Well, login as a UNIX user (root) should work.. My working assumption is that the problem is related in some way to the fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA 4.1, and switched between 2 FreeIPA servers. I am currently setting up 2 throwaway EL 7.1 VMs to better test this. On one I will first install 3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1 client. Cheers Chris From: Jakub Hrozek <jhro...@redhat.com> To: freeipa-users@redhat.com Date: 02.06.2015 09:22 Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Not Solved Sent by: freeipa-users-boun...@redhat.com On Mon, Jun 01, 2015 at 07:35:11PM +0200, Christopher Lamb wrote: Hi All Bad news. Over the weekend I was able to get the original problem EL7.1 / FreeIPA 4.1 host (FreeIPA client) to authenticate FreeiPA users (my test being ssh remote login with FreeIPA user and password). Today I tried a second machine, and had the same problem, ssh connections with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity check failed" This really just means wrong password, can you kinit as that user using the same password? Ahh I thought, I have a solution for that: just remove ipa-client and reinstall via yum, register with the new FreeIPA server .... Only with this second machine I still can't ssh in with a FreeIPA user. Argg..... b.t.w, as this machine is a real physical server, I was able to try logging in direct with my FreeIPA user --> "Authentication Failure" I now have * a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the old FreeIPA server to the new without a hitch (i.e. they successfully authenticate FreeIPA users.) * one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but with problems * one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts to authenticate with a FreeIPA user * one EL7.1 / FreeIPA 4.1 host that was only ever registered with the new FreeIPA server, and successfully authenticates FreeIPA users. Any ideas? Chris ----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17 ----- From: Christopher Lamb/Switzerland/IBM@IBMCH To: Alexander Bokovoy <aboko...@redhat.com>, freeipa-users@redhat.com Date: 30.05.2015 18:52 Subject: Re: [Freeipa-users] ssh problem with migrated FreeIPA client on EL7.1 --> Solved Sent by: freeipa-users-boun...@redhat.com Hi All It gives me pleasure to report the problem is solved - a minute ago I was able to login via ssh with my FreeIPA user to the problem server, while sitting on my terrace with a glass of wine! Thanks to Alexander for his helpful advice - we had some mail exchange outside the user list as I did not wish to broadcast content of keys, config files etc. Regardless of what I did with commands like klist, kvno everything seemed "ok", but I still could not ssh in. Even a ipa-getkeytab did not help. Therefore I decided to opt for brute force and (partial) ignorance. I completely uninstalled the FreeIPA client, and then reinstalled, configured - ét voilà I could ssh in! This leaves the enigma: what caused the problem? I suspect the following: The host is an EL 7.1, but the first FreeIPA client installed was version 3.3.3 (installed as set of standard packages that we bung on all our servers). This worked fine to authenticate against our "old" 3.x FreeIPA server, but did not work against the "new" 4.1 FreeIPA Server. When I realised I could not ssh in, one of the first things I did was to yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not help. The solution was to yum remove the FreeIPA client, then yum install the 4.1 client. I have some more EL 7.1 servers with the FreeIPA 3.3.3 client installed, so it will be interesting to see it the problem can be reproduced. Keep up the good work, Chris From: Alexander Bokovoy <aboko...@redhat.com> To: Christopher Lamb/Switzerland/IBM@IBMCH Cc: freeipa-users@redhat.com Date: 29.05.2015 18:04 Subject: Re: [Freeipa-users] ssh problem with migrated FreeIPA client on EL7.1 On Fri, 29 May 2015, Christopher Lamb wrote: Hi All Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to replace the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully migrated across the users. We have 50 odd Servers that are FreeIPA clients. Today I started migrating these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4 server by doing an ipa-client-install --uninstall from the old, and ipa-client-install to register with the new 4.1.0 server. Most of the FreeIPA clients are running OEL 6.5, and for these the migration process above worked perfectly. After migrating the server, I could ssh in with my FreeIPA user. Then I migrated an OEL 7.1 server. The migration itself seemed to work, and getent passwd was successful for my FreeIPA user. However when I try and ssh in, my FreeIPA user / password is not accepted. Before the migration I could ssh into the problem server (though evidently it was using my FreeIPA user from the old FreeIPA server). I can ssh in with a local (non ldap) user, so ssh is running and working. >From user root I can successfully su to my FreeIPA user. Further investigation showed that version of ipa-client installed was 3.3.3, so I yum updated this to 4.1.0. However I still cannot ssh into the OEL 7.1 box with my FreeIPA user. The same user continues to work for the 6.5 boxes. A colleague tried to ssh in with his FreeIPA user, and was also rejected, so the problem is not my user, but is probably for all FreeIPA users. A failed ssh login attempt causes the following error in /var/log/messages [sssd[krb5_child[5393]]]: Decrypt integrity check failed It means /etc/krb5.keytab contains keys from older system and SSSD picks them up. Can you show output of 'klist -kKet'? -- / Alexander Bokovoy -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- / Alexander Bokovoy -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project FYI https://fedorahosted.org/freeipa/ticket/5050 -- Thank you, Dmitri Pal Director of Engineering for IdM portfolio Red Hat, Inc.-- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project