[
https://issues.apache.org/jira/browse/SPARK-31559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Masiero Vanzin resolved SPARK-31559.
--------------------------------------------
Fix Version/s: 3.0.0
Assignee: Jungtaek Lim
Resolution: Fixed
> AM starts with initial fetched tokens in any attempt
> ----------------------------------------------------
>
> Key: SPARK-31559
> URL: https://issues.apache.org/jira/browse/SPARK-31559
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Fix For: 3.0.0
>
>
> The issue is only occurred in yarn-cluster mode.
> Submitter will obtain delegation tokens for yarn-cluster mode, and add these
> credentials to the launch context. AM will be launched with these
> credentials, and AM and driver are able to leverage these tokens.
> In Yarn cluster mode, driver is launched in AM, which in turn initializes
> token manager (while initializing SparkContext) and obtain delegation tokens
> (+ schedule to renew) if both principal and keytab are available.
> That said, even we provide principal and keytab to run application with
> yarn-cluster mode, AM always starts with initial tokens from launch context
> until token manager runs and obtains delegation tokens.
> So there's a "gap", and if user codes (driver) access to external system with
> delegation tokens (e.g. HDFS) before initializing SparkContext, it cannot
> leverage the tokens token manager will obtain. It will make the application
> fail if AM is killed "after" the initial tokens are expired and relaunched.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
