Hyukjin Kwon created SPARK-61973:
------------------------------------
Summary: Take SparkSession's artifact arguments as PySpark's:
upload each path of addArtifacts in turn, take its flags of their truth, and
take a Path-like local path in copyFromLocalToFs
Key: SPARK-61973
URL: https://issues.apache.org/jira/browse/SPARK-61973
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
SparkSession.addArtifacts (and addArtifact), SparkSession.copyFromLocalToFs,
and the client methods
that they share with SparkSession.client (add_artifacts, copy_from_local_to_fs)
differed from
PySpark's in three ways.
1. Paths, in turn. PySpark's SparkConnectClient.add_artifacts gives its
ArtifactManager one path at
a time (core.py:2461-2470): each path is parsed, its file opened, and
uploaded before the next
one is parsed. So when a later path fails, of a missing file or of a format
that the flags do
not take, the paths before it are uploaded and the server keeps them. The
drop-in parsed and
checked every path first, and uploaded none of them.
2. Flags. PySpark's addArtifacts refuses its flags with sum([file, pyfile,
archive]) > 1
(connect/session.py:1113), as Python adds and compares them, and its client
tests only their
truth (artifact.py:209-241). So an integer or another object is a flag, a
single flag of 2 is
refused, and a value that sum does not add raises its TypeError. The drop-in
took bools only.
3. Local paths. PySpark reads the local path of copyFromLocalToFs and
copy_from_local_to_fs with
Path(local_path).absolute() (artifact.py:257), so a str or an os.PathLike,
such as a pathlib.Path,
works. The drop-in took a str only.
Repro (Spark 4.3.0 server; /tmp/a.py and /tmp/n.txt exist, /tmp/missing.py does
not)
spark.addArtifacts("/tmp/a.py", "/tmp/missing.py", pyfile=True)
spark.addArtifacts("/tmp/n.txt", file=1)
spark.client.add_artifacts("/tmp/n.txt", pyfile=0, archive=0, file=1)
spark.addArtifacts("/tmp/a.py", pyfile=2)
spark.addArtifacts("/tmp/a.py", pyfile=None)
spark.copyFromLocalToFs(pathlib.Path("/tmp/n.txt"), "/dest/n.txt")
Actual (drop-in):
1. FileNotFoundError, and a.py is not uploaded: a UDF that imports it
raises ModuleNotFoundError.
2-5. TypeError: 'int' object is not an instance of 'bool' (and 'None' is
not an instance of 'bool').
6. TypeError: 'PosixPath' object is not an instance of 'str'.
Expected (pyspark-client 4.3.0):
1. FileNotFoundError, with a.py uploaded, which a UDF imports.
2-3. OK.
4. PySparkValueError INVALID_MULTIPLE_ARGUMENT_CONDITIONS.
5. TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'.
6. The file is sent; a server refuses a local destination unless
spark.sql.artifact.copyFromLocalToFs.allowDestLocal is set.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]