jam xu created SPARK-55077:
------------------------------
Summary: [CORE][K8S] Support
spark.kubernetes.archives.avoidDownloadSchemes for K8s Cluster Mode
Key: SPARK-55077
URL: https://issues.apache.org/jira/browse/SPARK-55077
Project: Spark
Issue Type: Improvement
Components: Kubernetes, Spark Core
Affects Versions: 4.0.0
Reporter: jam xu
What changes were proposed in this pull request?
Add support for `spark.kubernetes.archives.avoidDownloadSchemes`
configuration to avoid downloading archives to the driver in Kubernetes
cluster mode, similar to the existing
`spark.kubernetes.jars.avoidDownloadSchemes` (SPARK-47475).
Why are the changes needed?
When archives are large and executor count is high, downloading all
archives to the driver can cause network saturation and timeouts. This
feature allows executors to fetch archives directly from remote storage.
Does this PR introduce any user-facing change?
Yes. Add a new configuration:
`spark.kubernetes.archives.avoidDownloadSchemes`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
