Github user subrotosanyal commented on the issue:
https://github.com/apache/spark/pull/17335
hi @yaooqinn ,
This is a scenario where Spark is embed in client application (spark-client
mode).
In the method `Client#createContainerLaunchContext ()`, the
credentials(delegation tokens) obtained to run the spark application is added
to current `UserGroupInformation`(refer the the deleted line in the PR) which
shouldn't be the case. `UserGroupInformation` is a static global object which
once changed at any point of application id reflected throughout the JVM.
Further, the delegation tokens so added are also passed to the YARN platform
(specifically ResourceManager). Resource Manager expires the tokens of an
application after a certain period of time lead to expiration of the token
which is part of the Client which submitted the Spark Job.
The fix tries to remove the code where the Spark job specific credentials
are added to current `UserGroupInformation`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
