Hyukjin Kwon created SPARK-62071:
------------------------------------
Summary: Make the classic modules of the objects of a session
PySpark's, so that they are not instances of the classic classes, as in
pyspark-client
Key: SPARK-62071
URL: https://issues.apache.org/jira/browse/SPARK-62071
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The drop-in for pyspark-client 4.3.0 gives the objects of a session PySpark
Connect's classes, of the modules of pyspark.sql.connect (SPARK-61758,
SPARK-61852). SPARK-61852 made the classic classes of pyspark.sql.conf, tvf,
merge and observation PySpark's, vendored as they are. The classic names of the
other objects of a session still name the drop-in's Connect classes, so
`isinstance` against them is True, where pyspark-client gives False.
Repro, against a Spark 4.3.0 Connect server:
import pyspark.sql.readwriter, pyspark.sql.udtf
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
isinstance(spark.read, pyspark.sql.readwriter.DataFrameReader) # True;
pyspark-client: False
@F.udtf(returnType="a int")
class Echo:
def eval(self, a):
yield (a,)
isinstance(Echo, pyspark.sql.udtf.UserDefinedTableFunction) # True;
pyspark-client: False
pyspark.sql.readwriter.to_str # AttributeError; pyspark-client: PySpark's
function
The same holds for the classic Catalog, DataFrameWriter, DataFrameWriterV2,
DataStreamReader, DataStreamWriter, StreamingQueryManager, StreamingQuery,
UDFRegistration, UDTFRegistration, GroupedData and PandasCogroupedOps, which
pyspark.sql and pyspark.sql.streaming also name, and for the
UserDefinedFunction and UserDefinedTableFunction of pyspark.sql.functions.
The classic modules pyspark.sql.catalog, readwriter, udf, udtf and group,
pyspark.sql.pandas.group_ops, and pyspark.sql.streaming.readwriter and query
are shims of the drop-in, not PySpark's modules:
- pyspark.sql.readwriter has no to_str and no OptionUtils;
- pyspark.sql.group has no dfapi or df_varargs_api, and its GroupedData is not
a subclass of PandasGroupedOpsMixin;
- pyspark.sql.DataFrameReader(spark) and pyspark.sql.Catalog(spark) make the
drop-in's objects, and pyspark.sql.udf.UserDefinedFunction(f)(col) makes a
Column, where PySpark's classic classes raise on Spark Connect.
Expected, as in pyspark-client: the classic modules are PySpark's, of Apache
Spark 4.3.0 as they are, and the objects of a session are of PySpark Connect's
classes, of pyspark.sql.connect, which are not subclasses of the classic ones.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]