[ 
https://issues.apache.org/jira/browse/SPARK-61973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61973:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Take SparkSession's artifact arguments as PySpark's: upload each path of 
> addArtifacts in turn, take its flags of their truth, and take a Path-like 
> local path in copyFromLocalToFs
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61973
>                 URL: https://issues.apache.org/jira/browse/SPARK-61973
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> SparkSession.addArtifacts (and addArtifact), SparkSession.copyFromLocalToFs, 
> and the client methods
> that they share with SparkSession.client (add_artifacts, 
> copy_from_local_to_fs) differed from
> PySpark's in three ways.
> 1. Paths, in turn. PySpark's SparkConnectClient.add_artifacts gives its 
> ArtifactManager one path at
>    a time (core.py:2461-2470): each path is parsed, its file opened, and 
> uploaded before the next
>    one is parsed. So when a later path fails, of a missing file or of a 
> format that the flags do
>    not take, the paths before it are uploaded and the server keeps them. The 
> drop-in parsed and
>    checked every path first, and uploaded none of them.
> 2. Flags. PySpark's addArtifacts refuses its flags with sum([file, pyfile, 
> archive]) > 1
>    (connect/session.py:1113), as Python adds and compares them, and its 
> client tests only their
>    truth (artifact.py:209-241). So an integer or another object is a flag, a 
> single flag of 2 is
>    refused, and a value that sum does not add raises its TypeError. The 
> drop-in took bools only.
> 3. Local paths. PySpark reads the local path of copyFromLocalToFs and 
> copy_from_local_to_fs with
>    Path(local_path).absolute() (artifact.py:257), so a str or an os.PathLike, 
> such as a pathlib.Path,
>    works. The drop-in took a str only.
> Repro (Spark 4.3.0 server; /tmp/a.py and /tmp/n.txt exist, /tmp/missing.py 
> does not)
>     spark.addArtifacts("/tmp/a.py", "/tmp/missing.py", pyfile=True)
>     spark.addArtifacts("/tmp/n.txt", file=1)
>     spark.client.add_artifacts("/tmp/n.txt", pyfile=0, archive=0, file=1)
>     spark.addArtifacts("/tmp/a.py", pyfile=2)
>     spark.addArtifacts("/tmp/a.py", pyfile=None)
>     spark.copyFromLocalToFs(pathlib.Path("/tmp/n.txt"), "/dest/n.txt")
> Actual (drop-in):
>     1. FileNotFoundError, and a.py is not uploaded: a UDF that imports it 
> raises ModuleNotFoundError.
>     2-5. TypeError: 'int' object is not an instance of 'bool' (and 'None' is 
> not an instance of 'bool').
>     6. TypeError: 'PosixPath' object is not an instance of 'str'.
> Expected (pyspark-client 4.3.0):
>     1. FileNotFoundError, with a.py uploaded, which a UDF imports.
>     2-3. OK.
>     4. PySparkValueError INVALID_MULTIPLE_ARGUMENT_CONDITIONS.
>     5. TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'.
>     6. The file is sent; a server refuses a local destination unless
>        spark.sql.artifact.copyFromLocalToFs.allowDestLocal is set.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to