Re: [Freeipa-users] Kerberos Clock Skew too great

[Rakesh Rajasekharan]

I was seeing a lot of entries in the krb5kdc.log like below

"krb5kdc[10403](info): TGS_REQ (4 etypes {18 17 16 23}) 10.1.4.219: ISSUE:
authtime 1485450918, etypes {rep=18 tkt=18 ses=18}, host/my-host@MYDOMAIN"

On one env.. where users rarely log in... even there I see a lot of such
requests.


Finally , I think  I was able to track this down..  there are few local
accounts ( non freeipa ) on my hosts . These are used to run some custom
scripts through cron and run frequently ( every few mins ).
So, I feel  whenever thers a request for "su - " or a sudo to
the local user, that would also end up calling the Kerbros service.. and
since it runs so frequently on all the hosts.. they would be choking the
IPA master / replica with so many requests..

Please correct me If I am wrong in the above assumption.

Going by the above logic.. I have added filter_users section with these
users in the sssd.conf . Hopefully I would see a drop in the number of
requests




On Mon, Jan 23, 2017 at 11:27 PM, Robbie Harwood 
wrote:

> Rakesh Rajasekharan  writes:
>
> > one more question I was curious is.. when does the krb5kdc.log get
> entries
> > . .. I mean is it only when someone makes an attempt to login to a server
> > that the log file  krb5kdc.log on the IPA master gets updated or there
> are
> > other scenarios as well
>
> It's controlled by /etc/kdc.conf ; take a look at the "[logging]" section
> in
> `man 5 kdc.conf` for more information.
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Kerberos Clock Skew too great

[Robbie Harwood]

Rakesh Rajasekharan  writes:

> one more question I was curious is.. when does the krb5kdc.log get entries
> . .. I mean is it only when someone makes an attempt to login to a server
> that the log file  krb5kdc.log on the IPA master gets updated or there are
> other scenarios as well

It's controlled by /etc/kdc.conf ; take a look at the "[logging]" section in
`man 5 kdc.conf` for more information.


signature.asc
Description: PGP signature
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Kerberos Clock Skew too great

[Rakesh Rajasekharan]

thanks for the inputs..


one more question I was curious is.. when does the krb5kdc.log get entries
. .. I mean is it only when someone makes an attempt to login to a server
that the log file  krb5kdc.log on the IPA master gets updated or there are
other scenarios as well

Thanks
Rakesh

On Fri, Jan 20, 2017 at 3:09 AM, Robbie Harwood  wrote:

> Rakesh Rajasekharan  writes:
>
> >> Great, glad it's fixed!  Are these VMs?  If not, you may wish to
> >> (re?)configure automatic syncing.
> >
> > yes these are AWS instances. How do I reconfigure auto syncing . Is
> > there a documentation I can follow.
>
> During install of the IPA server, it will set up an NTP server (unless
> you ask it not to).  During enrollment of each IPA client, it will
> configure NTP against that server (unless you ask it not to).  Disabling
> it is the -N flag in both cases.
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Kerberos Clock Skew too great

[Robbie Harwood]

Rakesh Rajasekharan  writes:

>> Great, glad it's fixed!  Are these VMs?  If not, you may wish to
>> (re?)configure automatic syncing.
>
> yes these are AWS instances. How do I reconfigure auto syncing . Is
> there a documentation I can follow.

During install of the IPA server, it will set up an NTP server (unless
you ask it not to).  During enrollment of each IPA client, it will
configure NTP against that server (unless you ask it not to).  Disabling
it is the -N flag in both cases.


signature.asc
Description: PGP signature
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Kerberos Clock Skew too great

[Rakesh Rajasekharan]

Hi There,

Sorry could not get back on this  earlier,

> Great, glad it's fixed!  Are these VMs?  If not, you may wish to
> (re?)configure automatic syncing.
 yes these are AWS instances. How do  I reconfigure auto syncing . Is there
a documentation I can follow.
Sorry, haven't done this before and not much info on that part


Apart from this , I also have a correlation between the "Clock skew" issue
and an earlier issue that I posted in another thread.
Basically , noticed that whenver I see clock skew errors, I see a lot of
connections in SYNC_RECV state.

this is the list of SYNC_RECV connections

tcp0  0 10.0.8.45:88   10.0.30.49:42695SYN_RECV
tcp0  0 10.0.8.45:88   10.0.15.72:44991SYN_RECV
tcp0  0 10.0.8.45:88   10.0.2.82:53265 SYN_RECV
tcp0  0 10.0.8.45:88   10.0.31.253:57682   SYN_RECV
tcp0  0 10.0.8.45:88   10.0.34.208:53488   SYN_RECV
tcp0  0 10.0.8.45:88   10.0.27.17:47245SYN_RECV
tcp0  0 10.0.8.45:88   10.0.17.53:54504SYN_RECV
tcp0  0 10.0.8.45:88   10.0.24.78:47796SYN_RECV
tcp0  0 10.0.8.45:88   10.0.4.246:33607SYN_RECV
tcp0  0 10.0.8.45:88   10.0.27.91:34190SYN_RECV
tcp0  0 10.0.8.45:88   10.0.27.248:38012   SYN_RECV
tcp0  0 10.0.8.45:88   10.0.15.139:51319   SYN_RECV
tcp0  0 10.0.8.45:88   10.0.15.175:41188   SYN_RECV


Thanks,
Rakesh



On Tue, Jan 10, 2017 at 12:48 AM, Robbie Harwood 
wrote:

> Rakesh Rajasekharan  writes:
>
> > There were about 1500 hosts that were alerting for "clock skew" and the
> > issue went away only after I did a resync using ntpdate on all those
> hosts
>
> Great, glad it's fixed!  Are these VMs?  If not, you may wish to
> (re?)configure automatic syncing.
>
> > Is it possible that so many higher number of minor offsets adds up and
> > causes it. Coz from the individual offset it looks much below the 5min
> limit
>
> Not as such, if I understand you correctly?  This should only be a
> problem between any two machines that need to communicate (including the
> freeipa KDC).
>
> > Or, is there a way to tell whats the offset limit its actually looking
> for.
>
> 5 minutes almost certainly.  The parameter to configure it is
> "clockskew" in the config files, but I don't think IPA touches that.
>
> Hope that helps,
> --Robbie
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Kerberos Clock Skew too great

[Robbie Harwood]

Rakesh Rajasekharan  writes:

> There were about 1500 hosts that were alerting for "clock skew" and the
> issue went away only after I did a resync using ntpdate on all those hosts

Great, glad it's fixed!  Are these VMs?  If not, you may wish to
(re?)configure automatic syncing.

> Is it possible that so many higher number of minor offsets adds up and
> causes it. Coz from the individual offset it looks much below the 5min limit

Not as such, if I understand you correctly?  This should only be a
problem between any two machines that need to communicate (including the
freeipa KDC).

> Or, is there a way to tell whats the offset limit its actually looking for.

5 minutes almost certainly.  The parameter to configure it is
"clockskew" in the config files, but I don't think IPA touches that.

Hope that helps,
--Robbie


signature.asc
Description: PGP signature
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Kerberos Clock Skew too great

[Jakub Hrozek]

On Mon, Jan 09, 2017 at 02:07:21PM +0530, Rakesh Rajasekharan wrote:
> yes on the IPA server as well.. the offset isn't that high
> 
>  remote   refid  st t when poll reach   delay   offset
> jitter
> ==
> *ip-10-10-1-150.e 132.163.4.1012 u  119  128  3770.431   -0.279
> 0.348
> 
> So, my NTP server, the ipa client and the IPA master.. all seems to not
> have a high offset or a jitter.
> 
> There were about 1500 hosts that were alerting for "clock skew" and the
> issue went away only after I did a resync using ntpdate on all those hosts
> 
> Is it possible that so many higher number of minor offsets adds up and
> causes it. Coz from the individual offset it looks much below the 5min limit
> 
> Or, is there a way to tell whats the offset limit its actually looking for.

Sorry, I'm a bit out of my depth here, the only other suggestion I have
is to try kinit with KRB5_TRACE=/dev/stderr when that happens, which
should at least dump which KDC is the client talking to (if you have
multiple masters..)

> 
> Thanks,
> Rakesh
> 
> 
> 
> On Mon, Jan 9, 2017 at 1:42 PM, Jakub Hrozek  wrote:
> 
> > On Mon, Jan 09, 2017 at 01:07:06PM +0530, Rakesh Rajasekharan wrote:
> > > Hi,
> > >
> > > I am using a Freeipa 4.2.0 server.
> > >
> > > I sometimes see, "clock skew too great" errors in /var/log/krb5kdc.log.
> > And
> > > when this happens, usually logins or new ipa-cleint-install fails.
> > >
> > > When I checked on one of the hosts for which the clock skew was reported,
> > >
> > > #> ntpq -p
> > > remote   refid  st t when poll reach   delay   offset
> > > jitter
> > > 
> > ==
> > > *ip-10-10-1-150.e 171.66.97.1262 u  869 1024  3770.4480.047
> > > 0.142
> >
> > In general, 5 minutes is OK at least. But are you sure the server is also
> > in sync or just the client against an NTP server (iow, are you sure you
> > are checking the difference between a client and the KDC as well?)
> >
> > --
> > Manage your subscription for the Freeipa-users mailing list:
> > https://www.redhat.com/mailman/listinfo/freeipa-users
> > Go to http://freeipa.org for more info on the project
> >

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Kerberos Clock Skew too great

[Rakesh Rajasekharan]

yes on the IPA server as well.. the offset isn't that high

 remote   refid  st t when poll reach   delay   offset
jitter
==
*ip-10-10-1-150.e 132.163.4.1012 u  119  128  3770.431   -0.279
0.348

So, my NTP server, the ipa client and the IPA master.. all seems to not
have a high offset or a jitter.

There were about 1500 hosts that were alerting for "clock skew" and the
issue went away only after I did a resync using ntpdate on all those hosts

Is it possible that so many higher number of minor offsets adds up and
causes it. Coz from the individual offset it looks much below the 5min limit

Or, is there a way to tell whats the offset limit its actually looking for.

Thanks,
Rakesh



On Mon, Jan 9, 2017 at 1:42 PM, Jakub Hrozek  wrote:

> On Mon, Jan 09, 2017 at 01:07:06PM +0530, Rakesh Rajasekharan wrote:
> > Hi,
> >
> > I am using a Freeipa 4.2.0 server.
> >
> > I sometimes see, "clock skew too great" errors in /var/log/krb5kdc.log.
> And
> > when this happens, usually logins or new ipa-cleint-install fails.
> >
> > When I checked on one of the hosts for which the clock skew was reported,
> >
> > #> ntpq -p
> > remote   refid  st t when poll reach   delay   offset
> > jitter
> > 
> ==
> > *ip-10-10-1-150.e 171.66.97.1262 u  869 1024  3770.4480.047
> > 0.142
>
> In general, 5 minutes is OK at least. But are you sure the server is also
> in sync or just the client against an NTP server (iow, are you sure you
> are checking the difference between a client and the KDC as well?)
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Kerberos Clock Skew too great

[Jakub Hrozek]

On Mon, Jan 09, 2017 at 01:07:06PM +0530, Rakesh Rajasekharan wrote:
> Hi,
> 
> I am using a Freeipa 4.2.0 server.
> 
> I sometimes see, "clock skew too great" errors in /var/log/krb5kdc.log. And
> when this happens, usually logins or new ipa-cleint-install fails.
> 
> When I checked on one of the hosts for which the clock skew was reported,
> 
> #> ntpq -p
> remote   refid  st t when poll reach   delay   offset
> jitter
> ==
> *ip-10-10-1-150.e 171.66.97.1262 u  869 1024  3770.4480.047
> 0.142

In general, 5 minutes is OK at least. But are you sure the server is also
in sync or just the client against an NTP server (iow, are you sure you
are checking the difference between a client and the KDC as well?)

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Freeipa-users] Kerberos Clock Skew too great

[Rakesh Rajasekharan]

Hi,

I am using a Freeipa 4.2.0 server.

I sometimes see, "clock skew too great" errors in /var/log/krb5kdc.log. And
when this happens, usually logins or new ipa-cleint-install fails.

When I checked on one of the hosts for which the clock skew was reported,

#> ntpq -p
remote   refid  st t when poll reach   delay   offset
jitter
==
*ip-10-10-1-150.e 171.66.97.1262 u  869 1024  3770.4480.047
0.142


Does the above o/p looks fine interms of the ntp sync

Whats the max sync time difference thats allowed for a client.

Thanks
Rakesh
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project