sarutak opened a new pull request, #58871:
URL: https://github.com/apache/spark/pull/58871
### What changes were proposed in this pull request?
Add user-facing documentation for the OIDC credential propagation feature
(umbrella SPARK-57703). This is the documentation sub-task (SPARK-57901). It
documents the cloud-agnostic framework, its configuration, the security model,
the AWS reference provider, and Kubernetes usage, including how
provider-declared properties take effect on the driver as well as the executors.
- `docs/security.md`:
- Overview of OIDC credential propagation and its security model: the raw
identity token stays on the driver; only the short-lived, delegated service
credentials are propagated to executors. TLS/RPC-encryption recommendations.
- "How it works" walkthrough (token ingestion, per-scheme
`CredentialProvider` resolution, versioned distribution to executors, renewal).
Supported on YARN, Kubernetes, and local mode.
- Configuration table for the core `spark.security.oidc.*` keys (type,
default, description).
- "Custom CredentialProvider" and "AWS reference provider" sections (how
to enable and configure).
- "Provider-declared configuration and driver-side access":
provider-declared properties (`CredentialProvider.additionalSparkProperties()`)
are applied on both the driver and the executors (only for keys the user has
not set), so with no explicit provider wiring both access storage using the
propagated OIDC credentials. Documents two driver-side, early-startup
limitations -- access during `SparkContext` construction (`spark.jars` /
`spark.files` / `spark.archives` / `spark.checkpoint.dir` on a
`CredentialProvider`-served scheme run before credentials are resolved) and the
Hadoop `FileSystem` cache in Kubernetes cluster mode -- both recommending
`local://`.
- `docs/running-on-kubernetes.md`: an OIDC section with two complete
configuration examples -- workload-level ServiceAccount tokens and per-user
identity tokens (Kubernetes Secret mount) -- plus a cluster-mode caveat about
the `SparkSubmit` `FileSystem` cache.
- `docs/configuration.md` and `docs/cloud-integration.md`: cross-references
to the OIDC section.
### Why are the changes needed?
Users need documentation to configure and use OIDC credential propagation:
the configuration keys, the security model (the raw token is not sent to
executors), complete Kubernetes examples, and the driver-side behavior and its
early-startup limitations. Without this the feature is undocumented.
### Does this PR introduce _any_ user-facing change?
Yes -- documentation only (no behavior or API change).
### How was this patch tested?
Built the docs and reviewed the rendered Markdown; verified internal links
(e.g. the `running-on-kubernetes.md` caveat links to the `security.md`
subsection) and that all core `spark.security.oidc.*` keys are documented with
default and description.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
