Hyukjin Kwon created SPARK-60956:
------------------------------------
Summary: Upload the jars, Python files and archives of
addArtifacts in Spark Connect Rust as PySpark does
Key: SPARK-60956
URL: https://issues.apache.org/jira/browse/SPARK-60956
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{SparkSession.addArtifacts(*path, pyfile=False,
archive=False, file=False)}} uploads a {{.jar}} to {{jars/}}, and with
{{pyfile=True}} a Python file to {{pyfiles/}}, with {{archive=True}} an archive
to {{archives/}}, and with {{file=True}} a file to {{files/}}; any other file
raises {{UNSUPPORTED_OPERATION}}. The Rust-backed Python client ignores the
flags and uploads every path to {{files/}}, so a jar is not added to the
classpath, a Python file is not importable by UDFs, an archive is not
extracted, and {{addArtifacts("small.txt")}} succeeds. The Rust
{{SparkSession::add_artifacts}} uploads every path to {{files/}} too. Upload
them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
