Hyukjin Kwon created SPARK-61973:
------------------------------------

             Summary: Take SparkSession's artifact arguments as PySpark's: 
upload each path of addArtifacts in turn, take its flags of their truth, and 
take a Path-like local path in copyFromLocalToFs
                 Key: SPARK-61973
                 URL: https://issues.apache.org/jira/browse/SPARK-61973
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
SparkSession.addArtifacts (and addArtifact), SparkSession.copyFromLocalToFs, 
and the client methods
that they share with SparkSession.client (add_artifacts, copy_from_local_to_fs) 
differed from
PySpark's in three ways.

1. Paths, in turn. PySpark's SparkConnectClient.add_artifacts gives its 
ArtifactManager one path at
   a time (core.py:2461-2470): each path is parsed, its file opened, and 
uploaded before the next
   one is parsed. So when a later path fails, of a missing file or of a format 
that the flags do
   not take, the paths before it are uploaded and the server keeps them. The 
drop-in parsed and
   checked every path first, and uploaded none of them.
2. Flags. PySpark's addArtifacts refuses its flags with sum([file, pyfile, 
archive]) > 1
   (connect/session.py:1113), as Python adds and compares them, and its client 
tests only their
   truth (artifact.py:209-241). So an integer or another object is a flag, a 
single flag of 2 is
   refused, and a value that sum does not add raises its TypeError. The drop-in 
took bools only.
3. Local paths. PySpark reads the local path of copyFromLocalToFs and 
copy_from_local_to_fs with
   Path(local_path).absolute() (artifact.py:257), so a str or an os.PathLike, 
such as a pathlib.Path,
   works. The drop-in took a str only.

Repro (Spark 4.3.0 server; /tmp/a.py and /tmp/n.txt exist, /tmp/missing.py does 
not)
    spark.addArtifacts("/tmp/a.py", "/tmp/missing.py", pyfile=True)
    spark.addArtifacts("/tmp/n.txt", file=1)
    spark.client.add_artifacts("/tmp/n.txt", pyfile=0, archive=0, file=1)
    spark.addArtifacts("/tmp/a.py", pyfile=2)
    spark.addArtifacts("/tmp/a.py", pyfile=None)
    spark.copyFromLocalToFs(pathlib.Path("/tmp/n.txt"), "/dest/n.txt")
Actual (drop-in):
    1. FileNotFoundError, and a.py is not uploaded: a UDF that imports it 
raises ModuleNotFoundError.
    2-5. TypeError: 'int' object is not an instance of 'bool' (and 'None' is 
not an instance of 'bool').
    6. TypeError: 'PosixPath' object is not an instance of 'str'.
Expected (pyspark-client 4.3.0):
    1. FileNotFoundError, with a.py uploaded, which a UDF imports.
    2-3. OK.
    4. PySparkValueError INVALID_MULTIPLE_ARGUMENT_CONDITIONS.
    5. TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'.
    6. The file is sent; a server refuses a local destination unless
       spark.sql.artifact.copyFromLocalToFs.allowDestLocal is set.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to