Hyukjin Kwon created SPARK-62071:
------------------------------------

             Summary: Make the classic modules of the objects of a session 
PySpark's, so that they are not instances of the classic classes, as in 
pyspark-client
                 Key: SPARK-62071
                 URL: https://issues.apache.org/jira/browse/SPARK-62071
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The drop-in for pyspark-client 4.3.0 gives the objects of a session PySpark 
Connect's classes, of the modules of pyspark.sql.connect (SPARK-61758, 
SPARK-61852). SPARK-61852 made the classic classes of pyspark.sql.conf, tvf, 
merge and observation PySpark's, vendored as they are. The classic names of the 
other objects of a session still name the drop-in's Connect classes, so 
`isinstance` against them is True, where pyspark-client gives False.

Repro, against a Spark 4.3.0 Connect server:

    import pyspark.sql.readwriter, pyspark.sql.udtf
    from pyspark.sql import SparkSession, functions as F
    spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
    isinstance(spark.read, pyspark.sql.readwriter.DataFrameReader)  # True; 
pyspark-client: False

    @F.udtf(returnType="a int")
    class Echo:
        def eval(self, a):
            yield (a,)

    isinstance(Echo, pyspark.sql.udtf.UserDefinedTableFunction)  # True; 
pyspark-client: False
    pyspark.sql.readwriter.to_str  # AttributeError; pyspark-client: PySpark's 
function

The same holds for the classic Catalog, DataFrameWriter, DataFrameWriterV2, 
DataStreamReader, DataStreamWriter, StreamingQueryManager, StreamingQuery, 
UDFRegistration, UDTFRegistration, GroupedData and PandasCogroupedOps, which 
pyspark.sql and pyspark.sql.streaming also name, and for the 
UserDefinedFunction and UserDefinedTableFunction of pyspark.sql.functions.

The classic modules pyspark.sql.catalog, readwriter, udf, udtf and group, 
pyspark.sql.pandas.group_ops, and pyspark.sql.streaming.readwriter and query 
are shims of the drop-in, not PySpark's modules:
- pyspark.sql.readwriter has no to_str and no OptionUtils;
- pyspark.sql.group has no dfapi or df_varargs_api, and its GroupedData is not 
a subclass of PandasGroupedOpsMixin;
- pyspark.sql.DataFrameReader(spark) and pyspark.sql.Catalog(spark) make the 
drop-in's objects, and pyspark.sql.udf.UserDefinedFunction(f)(col) makes a 
Column, where PySpark's classic classes raise on Spark Connect.

Expected, as in pyspark-client: the classic modules are PySpark's, of Apache 
Spark 4.3.0 as they are, and the objects of a session are of PySpark Connect's 
classes, of pyspark.sql.connect, which are not subclasses of the classic ones.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to