Re: Expired tickets not renewed

[Victor Sudakov]

Nico Williams wrote:
> On Thu, Aug 10, 2017 at 09:24:08AM +0700, Victor Sudakov wrote:
> > 1. The 7.x kdc did not understand the heimdal.db Kerberos database
> > created by 1.5.2. Are they not compatible? What should I know about
> > this?
> 
> Looking at differences in lib/hdb/hdb.asn1... they should be compatible.
> Is it possible that in FreeBSD 7.4 uses a different DB type?

Now that I look into it, the pkg message when installing warns:


heimdal-7.1.0_1 uses a new database format which is incompatible
with Heimdal in the base system and heimdal-1.5.3_6 or prior.

Please read 20161112 in /usr/ports/UPDATING carefully to upgrade
your database.


And there is a copious upgrade manual: http://termbin.com/tatv

> 
> Can you dump/load?  That should work.

That's what is basically in the manual.

> 
> > 2. The utilities in the FreeBSD base system will remain linked to the
> > base system Heimdal libs (/usr/lib/libgssapi* instead of the newer
> > /usr/local/lib/libgssapi*). 
> 
> FreeBSD hasn't upgraded yet?  I thought it had.

It keeps a fairly recent version of Heimdal in the ports collection,
but does not upgrade Heimdal in the base system for some reason.

-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859

Re: Expired tickets not renewed

[Benjamin Kaduk]

On Wed, Aug 09, 2017 at 09:34:32PM -0500, Nico Williams wrote:
> 
> FreeBSD hasn't upgraded yet?  I thought it had.

Nobody was even willing to think about it until there was an official
release to upgrade to.  And now one of the likely suspects to do that
work is investigating an MIT krb5 import instead of doing the upgrade...

-Ben

Re: Expired tickets not renewed

[Nico Williams]

On Thu, Aug 10, 2017 at 09:24:08AM +0700, Victor Sudakov wrote:
> 1. The 7.x kdc did not understand the heimdal.db Kerberos database
> created by 1.5.2. Are they not compatible? What should I know about
> this?

Looking at differences in lib/hdb/hdb.asn1... they should be compatible.
Is it possible that in FreeBSD 7.4 uses a different DB type?

Can you dump/load?  That should work.

> 2. The utilities in the FreeBSD base system will remain linked to the
> base system Heimdal libs (/usr/lib/libgssapi* instead of the newer
> /usr/local/lib/libgssapi*). 

FreeBSD hasn't upgraded yet?  I thought it had.

Nico
-- 

Re: Expired tickets not renewed

[Victor Sudakov]

Roland C. Dowdeswell wrote:
> 
> > Now if I destroy the expired ticket by "kdestroy 
> > --credential=host/techno..."
> > a new ticket is received and gssapi-with-mic is again successful until
> > the new tickets expires again.
> > 
> > I'm beginning to think of a cron job which would destroy hourly all
> > the service tickets... all except the TGT.
> 
> To bring the conversation back a little to the original point:
> 
> It appears that Heimdal 1.5 had incorrect behaviour.  The ccache code
> should skip expired credentials when finding service tickets.  This looks
> like it was fixed by the following commit:
> 
>   commit 0f1ae2d10186afb654df8f50cc78663eb53f27a9
>   Author: Nicolas Williams 
>   Date:   Fri Aug 2 18:55:36 2013 -0500
> 
>   Use KRB5_TC_MATCH_TIMES when looking for creds
> 
> So, if you upgrade, this issue will be resolved.

That's good news. I could install the current Heimdal 7.4.0 from the
FreeBSD ports collection, however, there were two major problems
upgrading when I tried last time:

1. The 7.x kdc did not understand the heimdal.db Kerberos database
created by 1.5.2. Are they not compatible? What should I know about
this?

2. The utilities in the FreeBSD base system will remain linked to the
base system Heimdal libs (/usr/lib/libgssapi* instead of the newer
/usr/local/lib/libgssapi*). 


-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859

Re: Expired tickets not renewed

[Roland C. Dowdeswell]

On Wed, Aug 09, 2017 at 09:58:04PM +0700, Victor Sudakov wrote:
>

> Now if I destroy the expired ticket by "kdestroy --credential=host/techno..."
> a new ticket is received and gssapi-with-mic is again successful until
> the new tickets expires again.
> 
> I'm beginning to think of a cron job which would destroy hourly all
> the service tickets... all except the TGT.

To bring the conversation back a little to the original point:

It appears that Heimdal 1.5 had incorrect behaviour.  The ccache code
should skip expired credentials when finding service tickets.  This looks
like it was fixed by the following commit:

commit 0f1ae2d10186afb654df8f50cc78663eb53f27a9
Author: Nicolas Williams 
Date:   Fri Aug 2 18:55:36 2013 -0500

Use KRB5_TC_MATCH_TIMES when looking for creds

So, if you upgrade, this issue will be resolved.

-- 
Roland C. Dowdeswell

Re: Expired tickets not renewed

[Nico Williams]

On Wed, Aug 09, 2017 at 03:06:37PM -0400, Roland C. Dowdeswell wrote:
> It appears that Heimdal 1.5 had incorrect behaviour.  The ccache code
> should skip expired credentials when finding service tickets.  This looks
> like it was fixed by the following commit:
> 
>   commit 0f1ae2d10186afb654df8f50cc78663eb53f27a9
>   Author: Nicolas Williams 
>   Date:   Fri Aug 2 18:55:36 2013 -0500
> 
>   Use KRB5_TC_MATCH_TIMES when looking for creds
> 
> So, if you upgrade, this issue will be resolved.

Ahh, I should have recalled that I'd fixed that...  Thanks for finding
that.

Re: Expired tickets not renewed

[Nico Williams]

On Wed, Aug 09, 2017 at 03:01:16PM -0400, Jeffrey Altman wrote:
> I hope this is an unnecessary question, but will all Kerberos libraries
> that parse the file cache know to skip the expired entries and keep
> searching?  Or are there implementations that will only return the first
> service principal match?

The krb5 API used, krb5_cc_retrieve_cred(), supports this going back a
long time in MIT, and probably in Heimdal, but you have to ask for this
by including KRB5_TC_MATCH_TIMES in the options flags argument.

Nico
-- 

Re: Expired tickets not renewed

[Nico Williams]

On Wed, Aug 09, 2017 at 01:44:56PM -0500, Nico Williams wrote:
> We do need to re-think re-initialization in the new locking regimen --
> re-init via truncation probably works well enough right now, but mostly
> by accident.

Ah, right, we never do that.

Re: Expired tickets not renewed

[Nico Williams]

On Wed, Aug 09, 2017 at 06:34:27PM +, Viktor Dukhovni wrote:
> On Wed, Aug 09, 2017 at 01:11:07PM -0500, Nico Williams wrote:
> > On Wed, Aug 09, 2017 at 06:01:26PM +, Viktor Dukhovni wrote:
> > > On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:
> > > 
> > > > Btw, one of my ticket caches looks like this (probably MIT library):
> > > > 
> > > >   IssuedExpires   Principal
> > > > Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  
> > > > krbtgt/besserwisser@besserwisser.org
> > > > Aug  5 18:06:50 2017  >>>Expired<<< 
> > > > imap/jaja.besserwisser@besserwisser.org
> > > > Aug  6 18:35:34 2017  >>>Expired<<< 
> > > > imap/jaja.besserwisser@besserwisser.org
> > > > Aug  8 09:08:14 2017  >>>Expired<<< 
> > > > imap/jaja.besserwisser@besserwisser.org
> > > > Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  
> > > > imap/jaja.besserwisser@besserwisser.org
> > > > 
> > > > There is no reason that the expired service tickets are kept around,
> > > > but in this case, the code seems to keep them and append new tickets
> > > > at the end instead.
> > > 
> > > By design, the "FILE" credential cache type is *append-only*.  Much
> > > fancier read-write locking and recovery would be needed for a cache
> > > where entries can be deleted and replaced.
> > 
> > Actually, no, the FILE ccache does support deletion, certainly in
> > Heimdal 7.x.
> 
> That's in-place invalidation, not deletion.  I was talking about
> deletion.

Well, OK, but I'd call it logical deletion (I don't recall if we made
klist hide such entries, but if not we should).

Logical deletion is sufficient when it comes to making sure that we
fetch a new service ticket.  Actually, it's not even necessary, and I do
believe the library in 7.x will fetch a new ticket if there's only an
expired one in the ccache and an unexpired TGT is available.

Shifting subsequent entries over to *really* delete the old entry is
simply not possible with the new locking regimen.

Reusing logically-deleted slots is doable but tricky.

If the issue is ccache size, then one can always just reinitialize.  And
we should definitely consider doing that in krb5_get_credentials() and
friends to keep the ccache to a reasonable size.

We do need to re-think re-initialization in the new locking regimen --
re-init via truncation probably works well enough right now, but mostly
by accident.

Nico
-- 

Re: Expired tickets not renewed

[Nico Williams]

On Wed, Aug 09, 2017 at 02:25:11PM -0400, Roland C. Dowdeswell wrote:
> On Wed, Aug 09, 2017 at 01:11:07PM -0500, Nico Williams wrote:
> > Actually, no, the FILE ccache does support deletion, certainly in
> > Heimdal 7.x.
> 
> Well, we can invalidate entries but I don't think that we can re-use
> the slots because of locking issues.

We do not try to re-use them.

We could try to, but it'd be tricky and I'd rather not.

First, if a new entry is shorter than a candidate deleted entry then
we'd have to pad it.  That's easy enough, and if we couldn't do it we
wouldn't do it and that's that.

Second, if a write is interrupted (e.g., by SIGKILL) and we get a
partial write, then we'd have a corrupted entry.  Because we no longer
lock around reading, that would be bad.  We can recover from truncated
tail entries, but not from garbage entries in the middle.

The ccache entry deletion code might seem similarly unsafe, but it turns
out to be safe anyways if any part of it completes because of what it
does and the order in which things are written.

Nico
-- 

Re: Expired tickets not renewed

[Viktor Dukhovni]

On Wed, Aug 09, 2017 at 01:11:07PM -0500, Nico Williams wrote:
> On Wed, Aug 09, 2017 at 06:01:26PM +, Viktor Dukhovni wrote:
> > On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:
> > 
> > > Btw, one of my ticket caches looks like this (probably MIT library):
> > > 
> > >   IssuedExpires   Principal
> > > Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  
> > > krbtgt/besserwisser@besserwisser.org
> > > Aug  5 18:06:50 2017  >>>Expired<<< 
> > > imap/jaja.besserwisser@besserwisser.org
> > > Aug  6 18:35:34 2017  >>>Expired<<< 
> > > imap/jaja.besserwisser@besserwisser.org
> > > Aug  8 09:08:14 2017  >>>Expired<<< 
> > > imap/jaja.besserwisser@besserwisser.org
> > > Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  
> > > imap/jaja.besserwisser@besserwisser.org
> > > 
> > > There is no reason that the expired service tickets are kept around,
> > > but in this case, the code seems to keep them and append new tickets
> > > at the end instead.
> > 
> > By design, the "FILE" credential cache type is *append-only*.  Much
> > fancier read-write locking and recovery would be needed for a cache
> > where entries can be deleted and replaced.
> 
> Actually, no, the FILE ccache does support deletion, certainly in
> Heimdal 7.x.

That's in-place invalidation, not deletion.  I was talking about
deletion.

-- 
Viktor.

Re: Expired tickets not renewed

[Nico Williams]

On Wed, Aug 09, 2017 at 06:01:26PM +, Viktor Dukhovni wrote:
> On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:
> 
> > Btw, one of my ticket caches looks like this (probably MIT library):
> > 
> >   IssuedExpires   Principal
> > Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  
> > krbtgt/besserwisser@besserwisser.org
> > Aug  5 18:06:50 2017  >>>Expired<<< 
> > imap/jaja.besserwisser@besserwisser.org
> > Aug  6 18:35:34 2017  >>>Expired<<< 
> > imap/jaja.besserwisser@besserwisser.org
> > Aug  8 09:08:14 2017  >>>Expired<<< 
> > imap/jaja.besserwisser@besserwisser.org
> > Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  
> > imap/jaja.besserwisser@besserwisser.org
> > 
> > There is no reason that the expired service tickets are kept around,
> > but in this case, the code seems to keep them and append new tickets
> > at the end instead.
> 
> By design, the "FILE" credential cache type is *append-only*.  Much
> fancier read-write locking and recovery would be needed for a cache
> where entries can be deleted and replaced.

Actually, no, the FILE ccache does support deletion, certainly in
Heimdal 7.x.

Nico
-- 

Re: Expired tickets not renewed

[Viktor Dukhovni]

On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:

> Btw, one of my ticket caches looks like this (probably MIT library):
> 
>   IssuedExpires   Principal
> Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  
> krbtgt/besserwisser@besserwisser.org
> Aug  5 18:06:50 2017  >>>Expired<<< 
> imap/jaja.besserwisser@besserwisser.org
> Aug  6 18:35:34 2017  >>>Expired<<< 
> imap/jaja.besserwisser@besserwisser.org
> Aug  8 09:08:14 2017  >>>Expired<<< 
> imap/jaja.besserwisser@besserwisser.org
> Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  
> imap/jaja.besserwisser@besserwisser.org
> 
> There is no reason that the expired service tickets are kept around,
> but in this case, the code seems to keep them and append new tickets
> at the end instead.

By design, the "FILE" credential cache type is *append-only*.  Much
fancier read-write locking and recovery would be needed for a cache
where entries can be deleted and replaced.

-- 
Viktor.

Re: Expired tickets not renewed

[Harald Barth]

> debug1: Next authentication method: gssapi-with-mic
> debug1:  The context has expired

That looks to me like a bug where the library actually should try to
get a new service ticket from the TGT. I don't know if that works in
any heimdal libkrb as most often (at least in my use case) the TGT and
the service ticket have the same lifetime. I only use this scenario
when I use GSSAPI IMAP and there the application is linked against MIT
IIRC.

> Now if I destroy the expired ticket by "kdestroy --credential=host/techno..."
> a new ticket is received and gssapi-with-mic is again successful until
> the new tickets expires again.

Yes, then you create the same situation as with a freshly aquired TGT.

> I'm beginning to think of a cron job which would destroy hourly all
> the service tickets... all except the TGT.

Does the same problem happen with kgetcred host/techno instead of
ssh?

Do you have time to test this in newer heimdal?

Ugly workaround: A wrapper script could destroy all expired service
tickets.

Btw, one of my ticket caches looks like this (probably MIT library):

  IssuedExpires   Principal
Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  
krbtgt/besserwisser@besserwisser.org
Aug  5 18:06:50 2017  >>>Expired<<< 
imap/jaja.besserwisser@besserwisser.org
Aug  6 18:35:34 2017  >>>Expired<<< 
imap/jaja.besserwisser@besserwisser.org
Aug  8 09:08:14 2017  >>>Expired<<< 
imap/jaja.besserwisser@besserwisser.org
Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  
imap/jaja.besserwisser@besserwisser.org

There is no reason that the expired service tickets are kept around,
but in this case, the code seems to keep them and append new tickets
at the end instead.

Harald.

Re: Expired tickets not renewed

[Victor Sudakov]

Victor Sudakov wrote:
> > Against what gssapi library is your ssh linked 
> 
> Heimdal 1.5.2 from the FreeBSD 10.3 base system.
> 
> > and what does ssh -vvv
> > reveal why gssapi does not proceed?
> 
> Next time a service ticket expires, I'll post it here. But don't hold
> your breath, it's probably going to be something stupid like
> 'miscellaneous failure, see text'

Dear Harald,

Below is what "ssh -vvv" reveals:

debug3: receive packet: type 51
debug1: Authentications that can continue: 
publickey,gssapi-with-mic,keyboard-interactive
debug3: start over, passed a different list 
publickey,gssapi-with-mic,keyboard-interactive
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1:  The context has expired
Success

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/sudakov/.ssh/id_rsa

Now if I destroy the expired ticket by "kdestroy --credential=host/techno..."
a new ticket is received and gssapi-with-mic is again successful until
the new tickets expires again.

I'm beginning to think of a cron job which would destroy hourly all
the service tickets... all except the TGT.

-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859

Re: Expired tickets not renewed

[Harald Barth]

Against what gssapi library is your ssh linked and what does ssh -vvv
reveal why gssapi does not proceed?

Harald.

Expired tickets not renewed

[Victor Sudakov]

Dear Colleauges,

I have a perfectly valid TGT and some expired service tickets (see the
klist output below). When I ssh to techno2 or fs01, the ssh client
asks me for the password. Why does it happen? 

I expect that the ssh client should request a new service ticket for
the relevant host, and not just give up?


Credentials cache: FILE:/tmp/krb5cc_3001
Principal: sudakov@REALM.HIDDEN

  IssuedExpires   Principal
Aug  7 08:27:14 2017  Aug 14 08:27:14 2017  krbtgt/REALM.HIDDEN@REALM.HIDDEN
Aug  7 08:27:16 2017  >>>Expired<<< 
host/pager11.REALM.HIDDEN@REALM.HIDDEN
[...]
Aug  7 11:01:56 2017  >>>Expired<<< 
host/techno2.REALM.HIDDEN@REALM.HIDDEN
Aug  7 11:01:56 2017  >>>Expired<<< 
host/techno2.REALM.HIDDEN@REALM.HIDDEN
Aug  7 12:58:54 2017  >>>Expired<<< 
host/fs01-sibptus.REALM.HIDDEN@REALM.HIDDEN
Aug  7 12:58:54 2017  >>>Expired<<< 
host/fs01-sibptus.REALM.HIDDEN@REALM.HIDDEN


-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859