Hyukjin Kwon created SPARK-62061:
------------------------------------

             Summary: SparkSession.addArtifacts reads a path that is not a str, 
and pyspark.sql.connect.client exports its names, as PySpark Connect
                 Key: SPARK-62061
                 URL: https://issues.apache.org/jira/browse/SPARK-62061
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


A follow-up of SPARK-62023 (#863). Two differences of the client from 
pyspark-client 4.3.0:

1. A path of addArtifacts that is not a str. PySpark's 
SparkSession.addArtifacts (python/pyspark/sql/connect/session.py:1110-1121) 
checks its flags, then SparkConnectClient.add_artifacts 
(connect/client/core.py:2461-2470) reads, checks and uploads each path before 
the next, and ArtifactManager._parse_artifacts 
(connect/client/artifact.py:184-203) parses it with urllib.parse.urlparse. So:
   - spark.addArtifacts(pathlib.Path("/tmp/a.py"), pyfile=True), and 
client.add_artifacts(...) of one, raise AttributeError: 'PosixPath' object has 
no attribute 'decode' (and 'int' object of 5);
   - bytes, or a falsy object such as None or 0, parse into a scheme of bytes: 
PySparkRuntimeError [UNSUPPORTED_OPERATION] b'' scheme is not supported 
(b'file' of b"file:///tmp/a.py");
   - spark.addArtifacts(path, pyfile=True, archive=True) raises 
INVALID_MULTIPLE_ARGUMENT_CONDITIONS of any path, as the flags are checked 
first;
   - spark.addArtifacts("/tmp/a.py", pathlib.Path("/tmp/a.py"), pyfile=True) 
uploads the first before the second raises.
   The drop-in took the paths as a list of str, so each of these raised 
TypeError: 'PosixPath' object is not an instance of 'str', before the flags and 
before any upload.

2. The names of pyspark.sql.connect.client. PySpark's package is `from 
pyspark.sql.connect.client.core import *` and `from pyspark.sql.connect.logging 
import getLogLevel`, with no __all__. Its `from pyspark.sql.connect.client 
import *` gives ChannelBuilder, DefaultChannelBuilder, RpcDeadlines, 
SparkConnectClient, getLogLevel and the modules core, artifact, reattach and 
retries. AnalyzeResult, ConfigResult and PlanObservedMetrics are of 
pyspark.sql.connect.client.core only (core.py:647-798). The drop-in's package 
defined those three classes, so `from pyspark.sql.connect.client import 
AnalyzeResult` worked where PySpark raises ImportError, and its __all__ of the 
four classes made its `import *` give neither getLogLevel nor core.

Also checked, the same in both clients on master 80046f2f: a name that is not a 
str to registerJavaFunction, registerJavaUDAF, udtf.register and udf.register 
raises protobuf's TypeError "bad argument type for built-in operation" 
(SPARK-61911, SPARK-62023), and a UDF that is not Arrow-optimized raises the 
UNSUPPORTED_OPERATION of an abstract return type at the first request, not at 
the call (SPARK-61962 / #837).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to