I've been trying to debug a problem over the past couple days where some machines which have some long-running jobs writing to AFS seem to be losing their authentication...
These jobs are all running in PAGs, with a job renewing their tokens every so often. I even have that same job logging the output of 'tokens' to syslog to verify that there are tokens there, and they expire sometime in the way-future.
However, last night, on all of these machines, around the same time, they seem to have lost their tokens, and a "Tokens for AFS id... have expired" was logged to my syslog... Even though the daemon that was running in that process' PAG had logged a message just a few minutes earlier that the tokens were due to expire sometime the next day. This has been happening off and on for the past couple days, at around the same time on all of these machines, in the early morning hours... (this also doesn't make sense, as our default token life is 24 hours, so if perchance the process renewing tokens wasn't doing it's thing, you'd expect the tokens to expire about the time it was last started, right?)
Anyhow, I must say I've seen something like this before... In an instance where an AFS fileserver was "slow" responding to requests for various reasons, and calls that were in-flight to be processed hung out in the fileserver's receive queue for just a little too long, and by the time they were processed by the fileserver they were too 'old', causing the fileserver to throw an RX error back to all of the clients that were involved, causing them to think their tokens had expired and discard them... While this is a problem (no fileserver should be so backlogged that this happens), it's also quite a bother that a message from a single misbehaving fileserver to a client could cause the client to toss it's tokens just on it's good word...
I did some looking at the afs/afs_analyze.c code that deals with these states. Would there be some interest in making this handling a little smarter? Perhaps only discarding the tokens if they're *actually* expired, and instead allowing the call that returned the RX error to either fail or retry? (Also imagining it'd be darn nice to log what fileserver tossed back the error, because right now I'm at a loss on that end as well!)
-rob _______________________________________________ OpenAFS-devel mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-devel
