[ 
https://issues.apache.org/jira/browse/SPARK-62071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62071:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the classic modules of the objects of a session PySpark's, so that they 
> are not instances of the classic classes, as in pyspark-client
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62071
>                 URL: https://issues.apache.org/jira/browse/SPARK-62071
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> The drop-in for pyspark-client 4.3.0 gives the objects of a session PySpark 
> Connect's classes, of the modules of pyspark.sql.connect (SPARK-61758, 
> SPARK-61852). SPARK-61852 made the classic classes of pyspark.sql.conf, tvf, 
> merge and observation PySpark's, vendored as they are. The classic names of 
> the other objects of a session still name the drop-in's Connect classes, so 
> `isinstance` against them is True, where pyspark-client gives False.
> Repro, against a Spark 4.3.0 Connect server:
>     import pyspark.sql.readwriter, pyspark.sql.udtf
>     from pyspark.sql import SparkSession, functions as F
>     spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
>     isinstance(spark.read, pyspark.sql.readwriter.DataFrameReader)  # True; 
> pyspark-client: False
>     @F.udtf(returnType="a int")
>     class Echo:
>         def eval(self, a):
>             yield (a,)
>     isinstance(Echo, pyspark.sql.udtf.UserDefinedTableFunction)  # True; 
> pyspark-client: False
>     pyspark.sql.readwriter.to_str  # AttributeError; pyspark-client: 
> PySpark's function
> The same holds for the classic Catalog, DataFrameWriter, DataFrameWriterV2, 
> DataStreamReader, DataStreamWriter, StreamingQueryManager, StreamingQuery, 
> UDFRegistration, UDTFRegistration, GroupedData and PandasCogroupedOps, which 
> pyspark.sql and pyspark.sql.streaming also name, and for the 
> UserDefinedFunction and UserDefinedTableFunction of pyspark.sql.functions.
> The classic modules pyspark.sql.catalog, readwriter, udf, udtf and group, 
> pyspark.sql.pandas.group_ops, and pyspark.sql.streaming.readwriter and query 
> are shims of the drop-in, not PySpark's modules:
> - pyspark.sql.readwriter has no to_str and no OptionUtils;
> - pyspark.sql.group has no dfapi or df_varargs_api, and its GroupedData is 
> not a subclass of PandasGroupedOpsMixin;
> - pyspark.sql.DataFrameReader(spark) and pyspark.sql.Catalog(spark) make the 
> drop-in's objects, and pyspark.sql.udf.UserDefinedFunction(f)(col) makes a 
> Column, where PySpark's classic classes raise on Spark Connect.
> Expected, as in pyspark-client: the classic modules are PySpark's, of Apache 
> Spark 4.3.0 as they are, and the objects of a session are of PySpark 
> Connect's classes, of pyspark.sql.connect, which are not subclasses of the 
> classic ones.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to