[
https://issues.apache.org/jira/browse/SPARK-62071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62071:
------------------------------------
Assignee: Hyukjin Kwon
> Make the classic modules of the objects of a session PySpark's, so that they
> are not instances of the classic classes, as in pyspark-client
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62071
> URL: https://issues.apache.org/jira/browse/SPARK-62071
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> The drop-in for pyspark-client 4.3.0 gives the objects of a session PySpark
> Connect's classes, of the modules of pyspark.sql.connect (SPARK-61758,
> SPARK-61852). SPARK-61852 made the classic classes of pyspark.sql.conf, tvf,
> merge and observation PySpark's, vendored as they are. The classic names of
> the other objects of a session still name the drop-in's Connect classes, so
> `isinstance` against them is True, where pyspark-client gives False.
> Repro, against a Spark 4.3.0 Connect server:
> import pyspark.sql.readwriter, pyspark.sql.udtf
> from pyspark.sql import SparkSession, functions as F
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> isinstance(spark.read, pyspark.sql.readwriter.DataFrameReader) # True;
> pyspark-client: False
> @F.udtf(returnType="a int")
> class Echo:
> def eval(self, a):
> yield (a,)
> isinstance(Echo, pyspark.sql.udtf.UserDefinedTableFunction) # True;
> pyspark-client: False
> pyspark.sql.readwriter.to_str # AttributeError; pyspark-client:
> PySpark's function
> The same holds for the classic Catalog, DataFrameWriter, DataFrameWriterV2,
> DataStreamReader, DataStreamWriter, StreamingQueryManager, StreamingQuery,
> UDFRegistration, UDTFRegistration, GroupedData and PandasCogroupedOps, which
> pyspark.sql and pyspark.sql.streaming also name, and for the
> UserDefinedFunction and UserDefinedTableFunction of pyspark.sql.functions.
> The classic modules pyspark.sql.catalog, readwriter, udf, udtf and group,
> pyspark.sql.pandas.group_ops, and pyspark.sql.streaming.readwriter and query
> are shims of the drop-in, not PySpark's modules:
> - pyspark.sql.readwriter has no to_str and no OptionUtils;
> - pyspark.sql.group has no dfapi or df_varargs_api, and its GroupedData is
> not a subclass of PandasGroupedOpsMixin;
> - pyspark.sql.DataFrameReader(spark) and pyspark.sql.Catalog(spark) make the
> drop-in's objects, and pyspark.sql.udf.UserDefinedFunction(f)(col) makes a
> Column, where PySpark's classic classes raise on Spark Connect.
> Expected, as in pyspark-client: the classic modules are PySpark's, of Apache
> Spark 4.3.0 as they are, and the objects of a session are of PySpark
> Connect's classes, of pyspark.sql.connect, which are not subclasses of the
> classic ones.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]