Hi All I can now report back success (at least on my throwaway EL7.1 test VM).
To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC to a new FreeIPA 4.1 KDC 3 steps are required: 1) ipa-client-install --uninstall 2) rm -f /var/lib/sss/db/* 3) ipa-client-install --server ldap.my.example.com --domain my.example.com -N Having done this, my free-ipa user successfully authenticates (e.g. ssh remote login with free-ipa user / password To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required. Kudos and thanks go to Rob C for suggesting step 2. (Note that the directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as suggested earlier in this thread. Cheers Chris From: Martin Kosek <mko...@redhat.com> To: Christopher Lamb/Switzerland/IBM@IBMCH, freeipa-users@redhat.com Cc: Jakub Hrozek <jhro...@redhat.com>, Rob Crittenden <rcrit...@redhat.com> Date: 03.06.2015 10:39 Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Not Solved On 06/03/2015 10:30 AM, Christopher Lamb wrote: > Hi all > > This is a quick(ish) note to bring everybody up to speed on this issue. > Yesterday we had some private mail exchange on this issue as I did not wish > to broadcast the krb5 and ipa install logs to the user list. > > The basic situation is that we are in the process of migrating from an > FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As discussed > in a thread some weeks ago we did not do this by replicating (as perhaps we > should have done). Instead we migrated the users across. > > We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined to > the old KDC. We are now in the process of migrating these hosts to the new > 4.1 KDC. > > Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these joining > to the new KDC was trouble free, taking a few minutes each. After joining > the new KDC FreeIPA users authenticated properly. > > We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that were > joined direct to the new 4.1 KDC, never having been joined of the 3.3.3 > KDC. These were also trouble free. > > The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1 hosts > that were originally joined to the 3.3.3 KDC, and must be moved to join the > 4.1 KDC. These machines no longer authenticate valid FreeIPA users. I have > been able to reproduce this behaviour with a freshly setup VM joined first > to the 3.3.3 KDC, then moved to the 4.1 KDC. > > While the errors show in the krb5 child logs indicate that the password is > incorrect, the same user / password is happily accepted by all the other > hosts. > > It seems that in the process of moving / migrating the EL 7.1 / ipa-client > 4.1 from the old KDC to the new KDC, "something" is left behind that causes > problems. We have seen indications in the install logs that the kinit steps > called during ipa-client install are getting responses from the wrong (old) > KDC, and not from the new KDC. > > Frustratingly. over the weekend i managed to get one of the problem EL 7.1 > boxes to work. However I can't work out exactly what I was that I did that > did the trick. However it seems that some kind of major de-install / > cleanup + reinstall of the ipa-client may be needed. > > Rob has suggested that as part of such a cleanup I should do "rm > -f /var/lib/sssd/db/*". I will test this later today and report back. > > Thanks to Rob, Jakub, Martin, Alexander et al for their help and > suggestions so far. > > Chris Thanks for the background. The pain you are getting is exactly the reason why migration via replication to RHEL-7.1 is a better choice :-) Please let us know the result, I am curious how this works out. > > > > > From: Martin Kosek <mko...@redhat.com> > To: Christopher Lamb/Switzerland/IBM@IBMCH, > freeipa-users@redhat.com, Jakub Hrozek <jhro...@redhat.com> > Date: 03.06.2015 09:34 > Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA > client on EL7.1 -->Not Solved > > > > On 06/02/2015 06:15 PM, Christopher Lamb wrote: >> >> Hi >> >> Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause >> of this problem. Let's call them HOST09 and HOST10 >> >> Both are mimimum installs of EL7.1, with NTPD installed and configured. >> >> HOST09 had ipa-client 4.1 installed via yum, and was configured to use > our >> new FreeIPA 4.1 server, right from the start. --> My FreeIPA user >> authenticates successfully against this machine. >> >> HOST10 had ipa-client 4.1 installed as a dependency of one of our > standard >> config packages, and was first set to use our old FreeIPA 3.3.3 server. > --> >> My FreeIPA user authenticates successfully. against this machine. >> >> I then de-registered HOST10 from the FreeIPA 3.1 server, and registered >> against the new FreeIPA 4.1 server --> My FreeIPA users does NOT >> authenticate successfully. >> >> This replicates well the behaviour I saw with my production servers, > namely >> a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new > 4.1 >> FreeIPA server authenticate properly. >> >> b) EL 7.1 hosts with ipa-client 4.1 first registered against the old > 3.3.3 >> FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT >> authenticate properly >> >> Chris > > Hello, > > This is really strange. What I do not fully understand is what is the > "registration against a FreeIPA server". What server you install IPA client > should matter if the deployment is set up properly. The host enrollment > entry > should simply replicate to whole infrastructure. The only thing that will > probably differ is sssd.conf and krb5.conf as they will have different > primary > server set up, based on what your DNS setup is. > > It rather seems that the "reregistration" is what causes the issue. It > looks > like something cleanup problem during the process. I will let Jakub to help > here, I would suggest including the SSSD logs from the failed login, it may > help. > >> >> >> >> ----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52 >> ----- >> >> From: Christopher Lamb/Switzerland/IBM@IBMCH >> To: Jakub Hrozek <jhro...@redhat.com> >> Cc: freeipa-users@redhat.com >> Date: 02.06.2015 10:40 >> Subject: Re: [Freeipa-users] Fw: ssh problem >> with migrated > FreeIPA >> client on EL7.1 -->Not Solved >> Sent by: freeipa-users-boun...@redhat.com >> >> >> >> Hi Jakub >> >> Yes root login works, that's how I've been getting into the box. >> >> Surprisingly, kinit with my user seems to work on that box. After > entering >> my password when prompted, it returns to the commandline without error. >> >> However if I try kinit with another FreeIPA user, then instead of > prompting >> for a password, it gives "Generic preauthentication failure while getting >> initial credentials" error. >> >> Having set debug_level=10, when I try and ssh in with my FreeIPA user, I >> find errors like >> >> "Retrieving host .... with result: .. Matching credential not found" >> >> "Received error from KDC ... Additional pre-authentication required" >> >> "Received error from KDC... Decrypt integrity check failed" >> >> "Received error code 1432158219" >> >> Cheers >> >> Chris >> >> >> >> >> >> From: >> Jakub Hrozek <jhro...@redhat.com> >> To: Christopher Lamb/Switzerland/IBM@IBMCH >> Cc: freeipa-users@redhat.com >> Date: >> 02.06.2015 09:50 >> Subject: Re: [Freeipa-users] Fw: ssh problem with > migrated >> FreeIPA >> client on EL7.1 -->Not Solved >> >> >> >> On Tue, Jun 02, 2015 at 09:43:48AM +0200, Christopher Lamb wrote: >>> Hi Jakub >>> >>> The same user / password works with all our FreeIPA hosts - just this > one >>> box is the problem. So the password should be good. Of course a type is >>> always possible (especially for strong passwords), but I have tried many >>> times which should eliminate the odd password typo. The user / password >>> should also be good for both the old and the new FreeIPA Server. >> >> Interesting, can you add debug_level=10 to the domain section of >> sssd.conf? Then krb5_child.log should show Kerberos tracing info >> including which exact KDC SSSD was talking to. >> >>> >>> As I can neither log in direct, or via ssh to this box with my FreeIPA >>> user, I assume Kinit with my user won't work- i will try later in the >> day. >> >> Well, login as a UNIX user (root) should work.. >> >>> >>> My working assumption is that the problem is related in some way to the >>> fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA >>> 4.1, and switched between 2 FreeIPA servers. I am currently setting up 2 >>> throwaway EL 7.1 VMs to better test this. On one I will first install >>> 3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1 >>> client. >>> >>> Cheers >>> >>> Chris >>> >>> >>> >>> From: Jakub Hrozek > <jhro...@redhat.com> >>> To: > freeipa-users@redhat.com >>> Date: 02.06.2015 09:22 >>> Subject: Re: > [Freeipa-users] Fw: ssh problem with >> migrated >> FreeIPA >>> client on EL7.1 -->Not Solved >>> Sent by: > freeipa-users-boun...@redhat.com >>> >>> >>> >>> On Mon, Jun 01, 2015 at 07:35:11PM +0200, Christopher Lamb wrote: >>>> >>>> Hi All >>>> >>>> Bad news. >>>> >>>> Over the weekend I was able to get the original problem EL7.1 / FreeIPA >>> 4.1 >>>> host (FreeIPA client) to authenticate FreeiPA users (my test being ssh >>>> remote login with FreeIPA user and password). >>>> >>>> Today I tried a second machine, and had the same problem, ssh >> connections >>>> with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity >>> check >>>> failed" >>> >>> This really just means wrong password, can you kinit as that user using >>> the same password? >>> >>>> >>>> Ahh I thought, I have a solution for that: just remove ipa-client and >>>> reinstall via yum, register with the new FreeIPA server .... >>>> >>>> Only with this second machine I still can't ssh in with a FreeIPA user. >>>> Argg..... >>>> >>>> b.t.w, as this machine is a real physical server, I was able to try >>> logging >>>> in direct with my FreeIPA user --> "Authentication Failure" >>>> >>>> I now have >>>> * a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the >> old >>>> FreeIPA server to the new without a hitch (i.e. they successfully >>>> authenticate FreeIPA users.) >>>> * one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but >>>> with problems >>>> * one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts >>> to >>>> authenticate with a FreeIPA user >>>> * one EL7.1 / FreeIPA 4.1 host that was only ever registered with the >> new >>>> FreeIPA server, and successfully authenticates FreeIPA users. >>>> >>>> Any ideas? >>>> >>>> Chris >>>> >>>> >>>> ----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17 >>>> ----- >>>> >>>> From: > Christopher >> Lamb/Switzerland/IBM@IBMCH >>>> To: > Alexander > Bokovoy >> <aboko...@redhat.com>, >>>> freeipa-users@redhat.com >>>> Date: > 30.05.2015 > 18:52 >>>> Subject: > > Re: >> [Freeipa-users] ssh problem with >> migrated FreeIPA >>> client on >>>> EL7.1 --> Solved >>>> Sent by: >> freeipa-users-boun...@redhat.com >>>> >>>> >>>> >>>> Hi All >>>> >>>> It gives me pleasure to report the problem is solved - a minute ago I >> was >>>> able to login via ssh with my FreeIPA user to the problem server, while >>>> sitting on my terrace with a glass of wine! >>>> >>>> Thanks to Alexander for his helpful advice - we had some mail exchange >>>> outside the user list as I did not wish to broadcast content of keys, >>>> config files etc. >>>> >>>> Regardless of what I did with commands like klist, kvno everything >> seemed >>>> "ok", but I still could not ssh in. Even a ipa-getkeytab did not help. >>>> >>>> Therefore I decided to opt for brute force and (partial) ignorance. I >>>> completely uninstalled the FreeIPA client, and then reinstalled, >>> configured >>>> - ét voilà I could ssh in! >>>> >>>> This leaves the enigma: what caused the problem? I suspect the >> following: >>>> >>>> The host is an EL 7.1, but the first FreeIPA client installed was >> version >>>> 3.3.3 (installed as set of standard packages that we bung on all our >>>> servers). >>>> >>>> This worked fine to authenticate against our "old" 3.x FreeIPA server, >>> but >>>> did not work against the "new" 4.1 FreeIPA Server. >>>> >>>> When I realised I could not ssh in, one of the first things I did was >> to >>>> yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not >> help. >>>> The solution was to yum remove the FreeIPA client, then yum install the >>> 4.1 >>>> client. >>>> >>>> I have some more EL 7.1 servers with the FreeIPA 3.3.3 client >> installed, >>> so >>>> it will be interesting to see it the problem can be reproduced. >>>> >>>> Keep up the good work, >>>> >>>> Chris >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> From: >> Alexander Bokovoy >> <aboko...@redhat.com> >>>> To: >> Christopher >> Lamb/Switzerland/IBM@IBMCH >>>> Cc: >> freeipa-users@redhat.com >>>> Date: >> 29.05.2015 18:04 >>>> Subject: >> Re: >> [Freeipa-users] ssh problem with >>> migrated FreeIPA >>>> client on >>>> EL7.1 >>>> >>>> >>>> >>>> On Fri, 29 May 2015, Christopher Lamb wrote: >>>>> >>>>> Hi All >>>>> >>>>> Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to >>> replace >>>>> the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully >> migrated >>>>> across the users. >>>>> >>>>> We have 50 odd Servers that are FreeIPA clients. Today I started >>> migrating >>>>> these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4 >>>>> server by doing an ipa-client-install --uninstall from the old, and >>>>> ipa-client-install to register with the new 4.1.0 server. >>>>> >>>>> Most of the FreeIPA clients are running OEL 6.5, and for these the >>>>> migration process above worked perfectly. After migrating the server, >> I >>>>> could ssh in with my FreeIPA user. >>>>> >>>>> Then I migrated an OEL 7.1 server. The migration itself seemed to >> work, >>>> and >>>>> getent passwd was successful for my FreeIPA user. However when I try >> and >>>>> ssh in, my FreeIPA user / password is not accepted. >>>>> >>>>> Before the migration I could ssh into the problem server (though >>> evidently >>>>> it was using my FreeIPA user from the old FreeIPA server). >>>>> >>>>> I can ssh in with a local (non ldap) user, so ssh is running and >>> working. >>>>> >>>>> >From user root I can successfully su to my FreeIPA user. >>>>> >>>>> Further investigation showed that version of ipa-client installed was >>>>> 3.3.3, so I yum updated this to 4.1.0. >>>>> >>>>> However I still cannot ssh into the OEL 7.1 box with my FreeIPA user. >>> The >>>>> same user continues to work for the 6.5 boxes. >>>>> >>>>> A colleague tried to ssh in with his FreeIPA user, and was also >>> rejected, >>>>> so the problem is not my user, but is probably for all FreeIPA users. >>>>> >>>>> A failed ssh login attempt causes the following error >>> in /var/log/messages >>>>> >>>>> [sssd[krb5_child[5393]]]: Decrypt integrity check failed >>>> It means /etc/krb5.keytab contains keys from older system and SSSD >>>> picks them up. >>>> Can you show output of 'klist -kKet'? >>>> -- >>>> / Alexander Bokovoy >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Manage your subscription for the Freeipa-users mailing list: >>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>> Go to http://freeipa.org for more info on the project >>>> >>>> >>>> >>>> >>>> -- >>>> Manage your subscription for the Freeipa-users mailing list: >>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>> Go to http://freeipa.org for more info on the project >>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project >>> >>> >>> >>> >> >> >> >> >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project >> >> >> >> > > > > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project