malware27 opened a new pull request, #58406:
URL: https://github.com/apache/spark/pull/58406
### What changes were proposed in this pull request?
This PR updates Kubernetes dependency upload handling to preserve URI
fragment aliases for local dependencies.
For `spark.jars`, `spark.files`, `spark.archives`, and
`spark.submit.pyFiles`, it:
- Removes the fragment before uploading the physical file.
- Restores the fragment on the resulting uploaded URI.
- Preserves the original ordering of mixed local and remote dependencies.
- Leaves remote dependencies unchanged, including remote-only
configurations.
It also adds unit coverage for all four dependency configurations and a
Minikube integration test that verifies a `spark.files` alias is available from
both the driver working directory
and `SparkFiles` on executors.
### Why are the changes needed?
Spark supports dependency URIs in the form `source#alias`. For example:
```text
--files /tmp/query-random.sql#query.sql
The physical file should be uploaded as query-random.sql, while the
resulting spark.files entry should retain #query.sql.
On Kubernetes, locally uploaded files, JARs, and Python files lost this
fragment. Consequently, the driver could not access ./query.sql, and executors
could not resolve
SparkFiles.get("query.sql").
This differs from other cluster managers such as YARN, which preserve the
fragment as the application-visible localized name.
See SPARK-58969 (https://issues.apache.org/jira/browse/SPARK-58969).
### Does this PR introduce any user-facing change?
Yes.
Local Kubernetes dependencies using source#alias now retain their aliases
after upload. Applications can access the aliased name from the driver working
directory and through SparkFiles
on executors.
Dependencies without aliases and remote dependency URIs retain their
existing behavior.
### How was this patch tested?
Added 17 unit test cases covering:
- Aliased and unaliased local dependencies.
- JARs, files, archives, and Python files.
- Mixed local and remote dependency ordering.
- Duplicate local entries.
- Remote-only configurations.
- Empty and unset configurations.
- Verification that URI fragments are not passed to the physical Hadoop
upload.
Added a Minikube integration test for a local spark.files dependency with
an alias. The test verifies the alias in both the driver working directory and
executor SparkFiles.
The focused Maven test suite passed all 41 tests:
./build/mvn -Pkubernetes \
-pl resource-managers/kubernetes/core -am \
-DwildcardSuites=org.apache.spark.deploy.k8s.features.BasicDriverFeatureStepSuite
\
org.scalatest:scalatest-maven-plugin:2.2.0:test
Test compilation also passed:
./build/mvn -Pkubernetes \
-pl resource-managers/kubernetes/core -am \
-DskipTests test-compile
Scala formatting and style checks passed:
./dev/lint-scala
The Minikube integration test was not run locally because Minikube was
unavailable.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
