[
https://issues.apache.org/jira/browse/SPARK-61973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61973:
-----------------------------------
Labels: pull-request-available (was: )
> Take SparkSession's artifact arguments as PySpark's: upload each path of
> addArtifacts in turn, take its flags of their truth, and take a Path-like
> local path in copyFromLocalToFs
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61973
> URL: https://issues.apache.org/jira/browse/SPARK-61973
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> SparkSession.addArtifacts (and addArtifact), SparkSession.copyFromLocalToFs,
> and the client methods
> that they share with SparkSession.client (add_artifacts,
> copy_from_local_to_fs) differed from
> PySpark's in three ways.
> 1. Paths, in turn. PySpark's SparkConnectClient.add_artifacts gives its
> ArtifactManager one path at
> a time (core.py:2461-2470): each path is parsed, its file opened, and
> uploaded before the next
> one is parsed. So when a later path fails, of a missing file or of a
> format that the flags do
> not take, the paths before it are uploaded and the server keeps them. The
> drop-in parsed and
> checked every path first, and uploaded none of them.
> 2. Flags. PySpark's addArtifacts refuses its flags with sum([file, pyfile,
> archive]) > 1
> (connect/session.py:1113), as Python adds and compares them, and its
> client tests only their
> truth (artifact.py:209-241). So an integer or another object is a flag, a
> single flag of 2 is
> refused, and a value that sum does not add raises its TypeError. The
> drop-in took bools only.
> 3. Local paths. PySpark reads the local path of copyFromLocalToFs and
> copy_from_local_to_fs with
> Path(local_path).absolute() (artifact.py:257), so a str or an os.PathLike,
> such as a pathlib.Path,
> works. The drop-in took a str only.
> Repro (Spark 4.3.0 server; /tmp/a.py and /tmp/n.txt exist, /tmp/missing.py
> does not)
> spark.addArtifacts("/tmp/a.py", "/tmp/missing.py", pyfile=True)
> spark.addArtifacts("/tmp/n.txt", file=1)
> spark.client.add_artifacts("/tmp/n.txt", pyfile=0, archive=0, file=1)
> spark.addArtifacts("/tmp/a.py", pyfile=2)
> spark.addArtifacts("/tmp/a.py", pyfile=None)
> spark.copyFromLocalToFs(pathlib.Path("/tmp/n.txt"), "/dest/n.txt")
> Actual (drop-in):
> 1. FileNotFoundError, and a.py is not uploaded: a UDF that imports it
> raises ModuleNotFoundError.
> 2-5. TypeError: 'int' object is not an instance of 'bool' (and 'None' is
> not an instance of 'bool').
> 6. TypeError: 'PosixPath' object is not an instance of 'str'.
> Expected (pyspark-client 4.3.0):
> 1. FileNotFoundError, with a.py uploaded, which a UDF imports.
> 2-3. OK.
> 4. PySparkValueError INVALID_MULTIPLE_ARGUMENT_CONDITIONS.
> 5. TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'.
> 6. The file is sent; a server refuses a local destination unless
> spark.sql.artifact.copyFromLocalToFs.allowDestLocal is set.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]