Re: kvno X not found in keytab; ticket is likely out of date
> 3) anyway the best would be to pull old key from backups (either from > kdc or server backup) and put it back to KDC under correct kvno > > depending on your skills and other factors of your environment, > restoring whole KDC db might be easier than to mess with single entry ... btw, just putting old key to the service keytab on NFS server might do the trick most easily... the clients still holding the not-yet expired tickes would be able to access the service, because service would have both old and new keys available ... there should be no need to manage the kdc i guess b ps: typing faster than thinking ;( Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: kvno X not found in keytab; ticket is likely out of date
I'm definitely not an expert on the field, but I'd guess you'd have to: 1) wait until client tickets expires and clients requests new ones for current kvno 2) due to linux NFS credential storage burried deep in the kernel, reboot all clients (sometimes just restarting services helps, sometimes does not ;( 3) anyway the best would be to pull old key from backups (either from kdc or server backup) and put it back to KDC under correct kvno depending on your skills and other factors of your environment, restoring whole KDC db might be easier than to mess with single entry ... bodik Dne 07/22/2019 v 12:22 PM Laura Smith napsal(a): > Ok, I hold my hand up, I messed up. So the question is, how do I get myself > out of this mess ? > > A summary of how I got here: > • I have an NFS server and a bunch of clients connecting and auth using krb5. > • This was all working beautifully until today. > • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey > nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server > name instead of client name). > • Now everything is broken (none of the NFS clients can connect to the server > and I am seeing the error messages below on the NFS server). > • keytab on NFS server only had credentials for NFS server, so I deleted the > keytab and created a new one through ktadd > • that didnt' work. a reboot of the NFS server didn't work. > > Summary ? I'm up a smelly creek without a paddle ! > > Messages on NFS server: > > 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error > in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS > failure. Minor code may provide more information) - Request ticket server > nfs/foo.example@example.corp kvno 3 not found in keytab; ticket is likely > out of date > 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 > times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): > GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more > information) - Request ticket server nfs/foo.example@example.corp kvno 3 > not found in keytab; ticket is likely out of date] Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: kvno X not found in keytab; ticket is likely out of date
Probably. If I interpret your email, you recreated the key table for the server. I assume you either rebooted the server or restarted everything relevant (most critical would be rpc.svcgssd). I agree that rebooting clients would probably do it on the client side, except that not all systems are set up to clear /tmp on reboot. I don’t think that’s critical, but I can’t guarantee it. > On Jul 22, 2019, at 9:26 AM, Laura Smith > wrote: > > Hi Charles, > > Surely the action of rebooting the client would do all of that ? > > > ‐‐‐ Original Message ‐‐‐ > On Monday, July 22, 2019 2:13 PM, Charles Hedrick wrote: > >> Unfortunately it’s likely to take some experimentation. My starting point >> would be on each client, unmount the file system, maybe delete >> /tmp/krb5ccmachine*, restart rpc.gssd, and remount. >> >>> On Jul 22, 2019, at 6:22 AM, Laura Smith n5d9xq3ti233xiyif...@protonmail.ch >>> wrote: >>> Ok, I hold my hand up, I messed up. So the question is, how do I get myself >>> out of this mess ? >>> A summary of how I got here: >>> • I have an NFS server and a bunch of clients connecting and auth using >>> krb5. >>> • This was all working beautifully until today. >>> • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey >>> nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server >>> name instead of client name). >>> • Now everything is broken (none of the NFS clients can connect to the >>> server and I am seeing the error messages below on the NFS server). >>> • keytab on NFS server only had credentials for NFS server, so I deleted >>> the keytab and created a new one through ktadd >>> • that didnt' work. a reboot of the NFS server didn't work. >>> Summary ? I'm up a smelly creek without a paddle ! >>> Messages on NFS server: >>> 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: >>> error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE >>> (Unspecified GSS failure. Minor code may provide more information) - >>> Request ticket server nfs/foo.example@example.corp kvno 3 not found in >>> keytab; ticket is likely out of date >>> 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 >>> times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): >>> GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more >>> information) - Request ticket server nfs/foo.example@example.corp kvno >>> 3 not found in keytab; ticket is likely out of date] >>> >>> Kerberos mailing list Kerberos@mit.edu >>> https://mailman.mit.edu/mailman/listinfo/kerberos > > Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: kvno X not found in keytab; ticket is likely out of date
Hi Charles, Surely the action of rebooting the client would do all of that ? ‐‐‐ Original Message ‐‐‐ On Monday, July 22, 2019 2:13 PM, Charles Hedrick wrote: > Unfortunately it’s likely to take some experimentation. My starting point > would be on each client, unmount the file system, maybe delete > /tmp/krb5ccmachine*, restart rpc.gssd, and remount. > > > On Jul 22, 2019, at 6:22 AM, Laura Smith n5d9xq3ti233xiyif...@protonmail.ch > > wrote: > > Ok, I hold my hand up, I messed up. So the question is, how do I get myself > > out of this mess ? > > A summary of how I got here: > > • I have an NFS server and a bunch of clients connecting and auth using > > krb5. > > • This was all working beautifully until today. > > • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey > > nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server > > name instead of client name). > > • Now everything is broken (none of the NFS clients can connect to the > > server and I am seeing the error messages below on the NFS server). > > • keytab on NFS server only had credentials for NFS server, so I deleted > > the keytab and created a new one through ktadd > > • that didnt' work. a reboot of the NFS server didn't work. > > Summary ? I'm up a smelly creek without a paddle ! > > Messages on NFS server: > > 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: > > error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE > > (Unspecified GSS failure. Minor code may provide more information) - > > Request ticket server nfs/foo.example@example.corp kvno 3 not found in > > keytab; ticket is likely out of date > > 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 > > times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): > > GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more > > information) - Request ticket server nfs/foo.example@example.corp kvno > > 3 not found in keytab; ticket is likely out of date] > > > > Kerberos mailing list Kerberos@mit.edu > > https://mailman.mit.edu/mailman/listinfo/kerberos Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: kvno X not found in keytab; ticket is likely out of date
Unfortunately it’s likely to take some experimentation. My starting point would be on each client, unmount the file system, maybe delete /tmp/krb5ccmachine*, restart rpc.gssd, and remount. > On Jul 22, 2019, at 6:22 AM, Laura Smith > wrote: > > Ok, I hold my hand up, I messed up. So the question is, how do I get myself > out of this mess ? > > A summary of how I got here: > • I have an NFS server and a bunch of clients connecting and auth using krb5. > • This was all working beautifully until today. > • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey > nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server > name instead of client name). > • Now everything is broken (none of the NFS clients can connect to the server > and I am seeing the error messages below on the NFS server). > • keytab on NFS server only had credentials for NFS server, so I deleted the > keytab and created a new one through ktadd > • that didnt' work. a reboot of the NFS server didn't work. > > Summary ? I'm up a smelly creek without a paddle ! > > Messages on NFS server: > > 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error > in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS > failure. Minor code may provide more information) - Request ticket server > nfs/foo.example@example.corp kvno 3 not found in keytab; ticket is likely > out of date > 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 > times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): > GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more > information) - Request ticket server nfs/foo.example@example.corp kvno 3 > not found in keytab; ticket is likely out of date] > > > Kerberos mailing list Kerberos@mit.edu > https://mailman.mit.edu/mailman/listinfo/kerberos Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: kvno X not found in keytab; ticket is likely out of date
I'm not an expert but I'd try: 1) check if the keys for service are in sync in KDB and service keytab. if client reboot does not help, i'd guess keys are not in proper sync 2) pull old keytab from NFS server backup and merge it with current keytab client with not-yet expired tickets should be able to reconnect. service will use the old key for them, there's no communication with kdc until it those expires clients with new creds will use new key depending on the state of you environment, there might be some other issues regarding encryption types and problem that old clients cannot connect to the service which had old enctype, but now it has very new (due to the key regeneration) bodik Dne 07/22/2019 v 01:00 PM Laura Smith napsal(a): > Maybe a couple of hours or so. > > "klist -l" shows empty on a client I've tried. > > When mounting, the client now shows : > mount.nfs4: access denied by server while mounting (null) > mount: mounting foo.example.com:/srv/share/foo on /mnt/foo failed: Invalid > argument > > Demsg on the client shows: > NFS: state manager: check lease failed on NFSv4 server foo.example.com with > error 13 > > Rebooting the client has no effect. > > ‐‐‐ Original Message ‐‐‐ > On Monday, July 22, 2019 11:47 AM, John Devitofranceschi > wrote: > >> How long has it been since this happened? >> >> I think that the clients will be fine once their old ccaches expire. Have >> you tried forcing the issue by manually refreshing one of the clients? >> >> Sent from my iPhone >> >>> On Jul 22, 2019, at 06:22, Laura Smith n5d9xq3ti233xiyif...@protonmail.ch >>> wrote: >>> Ok, I hold my hand up, I messed up. So the question is, how do I get myself >>> out of this mess ? >>> A summary of how I got here: >>> • I have an NFS server and a bunch of clients connecting and auth using >>> krb5. >>> • This was all working beautifully until today. >>> • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey >>> nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server >>> name instead of client name). >>> • Now everything is broken (none of the NFS clients can connect to the >>> server and I am seeing the error messages below on the NFS server). >>> • keytab on NFS server only had credentials for NFS server, so I deleted >>> the keytab and created a new one through ktadd >>> • that didnt' work. a reboot of the NFS server didn't work. >>> Summary ? I'm up a smelly creek without a paddle ! >>> Messages on NFS server: >>> 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: >>> error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE >>> (Unspecified GSS failure. Minor code may provide more information) - >>> Request ticket server nfs/foo.example@example.corp kvno 3 not found in >>> keytab; ticket is likely out of date >>> 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 >>> times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): >>> GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more >>> information) - Request ticket server nfs/foo.example@example.corp kvno >>> 3 not found in keytab; ticket is likely out of date] >>> >>> Kerberos mailing list Kerberos@mit.edu >>> https://mailman.mit.edu/mailman/listinfo/kerberos > > > > > Kerberos mailing list Kerberos@mit.edu > https://mailman.mit.edu/mailman/listinfo/kerberos > Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: kvno X not found in keytab; ticket is likely out of date
Maybe a couple of hours or so. "klist -l" shows empty on a client I've tried. When mounting, the client now shows : mount.nfs4: access denied by server while mounting (null) mount: mounting foo.example.com:/srv/share/foo on /mnt/foo failed: Invalid argument Demsg on the client shows: NFS: state manager: check lease failed on NFSv4 server foo.example.com with error 13 Rebooting the client has no effect. ‐‐‐ Original Message ‐‐‐ On Monday, July 22, 2019 11:47 AM, John Devitofranceschi wrote: > How long has it been since this happened? > > I think that the clients will be fine once their old ccaches expire. Have you > tried forcing the issue by manually refreshing one of the clients? > > Sent from my iPhone > > > On Jul 22, 2019, at 06:22, Laura Smith n5d9xq3ti233xiyif...@protonmail.ch > > wrote: > > Ok, I hold my hand up, I messed up. So the question is, how do I get myself > > out of this mess ? > > A summary of how I got here: > > • I have an NFS server and a bunch of clients connecting and auth using > > krb5. > > • This was all working beautifully until today. > > • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey > > nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server > > name instead of client name). > > • Now everything is broken (none of the NFS clients can connect to the > > server and I am seeing the error messages below on the NFS server). > > • keytab on NFS server only had credentials for NFS server, so I deleted > > the keytab and created a new one through ktadd > > • that didnt' work. a reboot of the NFS server didn't work. > > Summary ? I'm up a smelly creek without a paddle ! > > Messages on NFS server: > > 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: > > error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE > > (Unspecified GSS failure. Minor code may provide more information) - > > Request ticket server nfs/foo.example@example.corp kvno 3 not found in > > keytab; ticket is likely out of date > > 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 > > times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): > > GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more > > information) - Request ticket server nfs/foo.example@example.corp kvno > > 3 not found in keytab; ticket is likely out of date] > > > > Kerberos mailing list Kerberos@mit.edu > > https://mailman.mit.edu/mailman/listinfo/kerberos Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: kvno X not found in keytab; ticket is likely out of date
How long has it been since this happened? I think that the clients will be fine once their old ccaches expire. Have you tried forcing the issue by manually refreshing one of the clients? Sent from my iPhone > On Jul 22, 2019, at 06:22, Laura Smith > wrote: > > Ok, I hold my hand up, I messed up. So the question is, how do I get myself > out of this mess ? > > A summary of how I got here: > • I have an NFS server and a bunch of clients connecting and auth using krb5. > • This was all working beautifully until today. > • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey > nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server > name instead of client name). > • Now everything is broken (none of the NFS clients can connect to the server > and I am seeing the error messages below on the NFS server). > • keytab on NFS server only had credentials for NFS server, so I deleted the > keytab and created a new one through ktadd > • that didnt' work. a reboot of the NFS server didn't work. > > Summary ? I'm up a smelly creek without a paddle ! > > Messages on NFS server: > > 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error > in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS > failure. Minor code may provide more information) - Request ticket server > nfs/foo.example@example.corp kvno 3 not found in keytab; ticket is likely > out of date > 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 > times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): > GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more > information) - Request ticket server nfs/foo.example@example.corp kvno 3 > not found in keytab; ticket is likely out of date] > > > Kerberos mailing list Kerberos@mit.edu > https://mailman.mit.edu/mailman/listinfo/kerberos smime.p7s Description: S/MIME cryptographic signature Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
