Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17335
So I had to dig up some e-mails to refresh my brain about SPARK-15754. It
is not related to YARN-2964 (that one is for things like Oozie, where the same
token is using by multiple YARN apps). It's related to YARN cancelling tokens
after apps finish (or a group of apps sharing the same token, in case of
YARN-2964).
So, in the embedded case, something like this:
```
val sc1 = new SparkContext("yarn-client")
// do stuff
sc1.stop()
// wait a bit
// The following will fail because YARN will have cancelled the old
delegation tokens
// which are still in the current UGI object.
val sc2 = new SparkContext("yarn-client")
```
The problem is caused by Spark adding the tokens to the current UGI, and
the UGI API has no way to remove them. So when you start the new context, the
code will try to use the tokens in the UGI and fail because they've been
cancelled.
Allowing Spark to overwrite the current UGI's credentials seems to fix a
bunch of issues, and is obviously fine for everybody using `spark-submit`. But
I wonder if there's a way to avoid that in these applications that embed Spark
without requiring them to manage their own delegation tokens.
Let me dig up some code from my e-mail and see if I can reproduce the
original issue and find a workaround...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
