Kousuke Saruta created SPARK-57894:
--------------------------------------
Summary: Implement UpdateUserCredentials RPC, backend integration,
and task closure delivery
Key: SPARK-57894
URL: https://issues.apache.org/jira/browse/SPARK-57894
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 4.3.0
Reporter: Kousuke Saruta
This sub-task is part of the umbrella
[SPARK-57703|https://issues.apache.org/jira/browse/SPARK-57703] (OIDC
Credential Propagation).
h2. Problem
There is no RPC to distribute delegated credentials from the driver to
executors, and no guarantee that credentials are available before task
execution.
h2. Goal
Add credential distribution via RPC broadcast and task closure, ensuring
executors always have valid credentials when running tasks.
h2. Scope
* New message:
{code:scala}
case class UpdateUserCredentials(payload: Array[Byte]) extends
CoarseGrainedClusterMessage
{code}
* {{CoarseGrainedSchedulerBackend}}: new method
{{updateUserCredentials(bytes)}} that stores in {{AtomicReference}} and sends
to all registered executors. Wired to {{UserCredentialManager}} callback.
* {{CoarseGrainedExecutorBackend}}: handles {{UpdateUserCredentials}} by
updating executor credential store.
* {{SparkEnv}}: add credential store field.
* {{TaskDescription}}: include current credentials to guarantee availability
before task execution (avoids race between RPC broadcast and task dispatch).
* RPC broadcast remains for credential refresh during long-running tasks.
h2. Acceptance criteria
* {{UpdateUserCredentials}} is sent to all executors on credential update.
* Executor credential store is updated on receiving the RPC.
* {{TaskDescription}} contains current credentials; executor applies them
before task runs.
* rawToken is NOT present in any RPC payload or TaskDescription.
* Unit tests for RPC encoding/decoding, backend handling, and task closure
credential verification.
Reference: [SPIP design document Appendix
B|https://docs.google.com/document/d/1usJKncCPMiyFUg7aIdpZ0HQsklXIHow_sU_6dfFMjN0/edit].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
