Kishan Ghetia created SPARK-58969:
-------------------------------------
Summary: [K8S] Local dependency uploads lose URI fragment aliases
Key: SPARK-58969
URL: https://issues.apache.org/jira/browse/SPARK-58969
Project: Spark
Issue Type: Bug
Components: Bug
Affects Versions: 4.1.2
Reporter: Kishan Ghetia
Spark's Kubernetes submission path drops URI fragments from locally uploaded
dependencies.
Spark accepts dependency URIs in the form `source#alias`. The fragment defines
the application-visible name. For example:
--files /tmp/query-random.sql#query.sql
The physical object should be uploaded as `query-random.sql`, while the
resulting `spark.files` entry should retain `#query.sql`. This allows the
driver to open `./query.sql` and executors to resolve
`SparkFiles.get("query.sql")`.
On Kubernetes, `BasicDriverFeatureStep` passes local FILES, JARS, and
SUBMIT_PYTHON_FILES through `uploadAndTransformFileUris`, but does not restore
their fragments afterward. Archive fragments are handled separately and do not
have this problem.
This differs from YARN, where the distributed-cache implementation retains the
URI fragment and uses it as the localized link name.
The issue can be reproduced by running this test -
[https://github.com/malware27/spark/commit/0fce035c761a17555b16b16d22141ac3a683e81a]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
