Github user ArtRand commented on the issue: https://github.com/apache/spark/pull/19272 Hey @kalvinnchau thanks for having the patience to try this. This is a curious error though. If you look at the `addAll` method called by `UserGroupInformation.addCredentials()` it should overwrite the current credentials. I tried to reproduce your error, but being less patient, I changed my HDFS setup to request the tokens be updated every minute instead of everyday by adding the following to hdfs-site.xml: ``` <property> <name>dfs.namenode.delegation.token.max-lifetime</name> <value>60000</value> </property> ``` I added some logging to the executor backend to check if they were indeed being updated. ``` case UpdateDelegationTokens(tokens) => logInfo("Got request to update tokens") val oldCreds = UserGroupInformation.getCurrentUser.getCredentials for (t <- oldCreds.getAllTokens.asScala) { logInfo(s"Old Creds ${DelegationTokenIdentifier.stringifyToken(t)}") } val creds = SparkHadoopUtil.get.deserialize(tokens) for (t <- creds.getAllTokens.asScala) { val s = DelegationTokenIdentifier.stringifyToken(t) logInfo(s"Got new tokens $s") } SparkHadoopUtil.get.addDelegationTokens(tokens, env.conf) val newCreds = UserGroupInformation.getCurrentUser.getCredentials for (t <- newCreds.getAllTokens.asScala) { logInfo(s"New creds ${DelegationTokenIdentifier.stringifyToken(t)}") } ``` and indeed when I check the logs, indeed the token number has been updated. ``` 17/09/28 03:32:58 INFO CoarseGrainedExecutorBackend: Got request to update tokens 17/09/28 03:32:58 INFO CoarseGrainedExecutorBackend: Old Creds HDFS_DELEGATION_TOKEN token 29 for hdfs on ha-hdfs:hdfs 17/09/28 03:32:59 INFO CoarseGrainedExecutorBackend: Got new tokens HDFS_DELEGATION_TOKEN token 31 for hdfs on ha-hdfs:hdfs 17/09/28 03:32:59 INFO CoarseGrainedExecutorBackend: New creds HDFS_DELEGATION_TOKEN token 31 for hdfs on ha-hdfs:hdfs ``` then some time later (in fact there was another update in the middle): ``` 17/09/28 03:35:14 INFO CoarseGrainedExecutorBackend: Got request to update tokens 17/09/28 03:35:14 INFO CoarseGrainedExecutorBackend: Old Creds HDFS_DELEGATION_TOKEN token 34 for hdfs on ha-hdfs:hdfs 17/09/28 03:35:14 INFO CoarseGrainedExecutorBackend: Got new tokens HDFS_DELEGATION_TOKEN token 35 for hdfs on ha-hdfs:hdfs 17/09/28 03:35:14 INFO CoarseGrainedExecutorBackend: New creds HDFS_DELEGATION_TOKEN token 35 for hdfs on ha-hdfs:hdfs ``` I will run a 24h experiment to verify, but hopefully there is a way to validate that the update is working without waiting that long just to debug! @vanzin Could you eyeball this, am I missing something obvious?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org