Jungtaek Lim created SPARK-31559:
------------------------------------
Summary: AM starts with initial fetched tokens in any attempt
Key: SPARK-31559
URL: https://issues.apache.org/jira/browse/SPARK-31559
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 3.0.0
Reporter: Jungtaek Lim
The issue is only occurred in yarn-cluster mode.
Submitter will obtain delegation tokens for yarn-cluster mode, and add these
credentials to the launch context. AM will be launched with these credentials,
and AM and driver are able to leverage these tokens.
In Yarn cluster mode, driver is launched in AM, which in turn initializes token
manager (while initializing SparkContext) and obtain delegation tokens (+
schedule to renew) if both principal and keytab are available.
That said, even we provide principal and keytab to run application with
yarn-cluster mode, AM always starts with initial tokens from launch context
until token manager runs and obtains delegation tokens.
So there's a "gap", and if user codes (driver) access to external system with
delegation tokens (e.g. HDFS) before initializing SparkContext, it cannot
leverage the tokens token manager will obtain. It will make the application
fail if AM is killed "after" the initial tokens are expired and relaunched.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
