Hyukjin Kwon created SPARK-62084:
------------------------------------
Summary: Make pyspark.sql.Column, DataFrameNaFunctions,
DataFrameStatFunctions, Window, WindowSpec and TableArg PySpark's classic
classes, of which the native Connect classes are virtual subclasses, as in
pyspark-client
Key: SPARK-62084
URL: https://issues.apache.org/jira/browse/SPARK-62084
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The drop-in for pyspark-client 4.3.0 gives the objects of a session PySpark
Connect's classes. After SPARK-62071 (the classic modules of the objects of a
session made PySpark's), five classic names still name the drop-in's native
Connect classes, and the classic Window is a native class too:
- pyspark.sql.Column (pyspark.sql.column.Column) is
pyspark.sql.connect.column.Column;
- pyspark.sql.DataFrameNaFunctions and DataFrameStatFunctions are the Connect
classes of pyspark.sql.connect.dataframe;
- pyspark.sql.WindowSpec is pyspark.sql.connect.window.WindowSpec;
- pyspark.sql.table_arg.TableArg is pyspark.sql.connect.table_arg.TableArg;
- pyspark.sql.window.Window is a native class, not PySpark's classic class.
In PySpark 4.3.0, each classic class is a separate Python class, which its
Connect class subclasses (python/pyspark/sql/connect/column.py: class
Column(ParentColumn); connect/dataframe.py, connect/window.py,
connect/table_arg.py likewise).
Repro, against a Spark 4.3.0 Connect server:
import pyspark.sql.column, pyspark.sql.connect.column
from pyspark.sql import Window, functions as F
pyspark.sql.Column is pyspark.sql.connect.column.Column # True;
pyspark-client: False
pyspark.sql.Column.__mro__ # (connect Column, object);
pyspark-client: (Column, TableValuedFunctionArgument, object)
pyspark.sql.Column.__module__ # 'pyspark.sql.connect.column';
pyspark-client: 'pyspark.sql.column'
pyspark.sql.Column.alias(object(), "x") # TypeError of a native
descriptor; pyspark-client: NOT_IMPLEMENTED (dispatch_col_method)
class MyColumn(pyspark.sql.Column): pass # TypeError: type 'Column' is
not an acceptable base type; pyspark-client: a subclass
pyspark.sql.DataFrameNaFunctions(df) # TypeError: cannot create ...
instances; pyspark-client: an object of df
pyspark.sql.dataframe.DataFrameNaFunctions.drop(df) # TypeError;
pyspark-client: df.drop() (dispatch_df_method)
The same holds for DataFrameStatFunctions, WindowSpec and TableArg; and
pyspark.sql.tvf_argument.TableValuedFunctionArgument is an abc.ABC, of ABC in
its MRO, where PySpark's is a plain class.
Expected, as in pyspark-client: the classic classes are PySpark's
(python/pyspark/sql/column.py, table_arg.py, window.py and the classes of
dataframe.py), of their modules, names, MROs, docstrings and dispatching
methods, and the objects of a session are of the Connect classes, which are
subclasses of them (for the drop-in's native classes, virtual ones), so that
isinstance and issubclass hold, and a patch of a classic class reaches the
objects as it does through PySpark's MRO.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]