Hyukjin Kwon created SPARK-62084:
------------------------------------

             Summary: Make pyspark.sql.Column, DataFrameNaFunctions, 
DataFrameStatFunctions, Window, WindowSpec and TableArg PySpark's classic 
classes, of which the native Connect classes are virtual subclasses, as in 
pyspark-client
                 Key: SPARK-62084
                 URL: https://issues.apache.org/jira/browse/SPARK-62084
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The drop-in for pyspark-client 4.3.0 gives the objects of a session PySpark 
Connect's classes. After SPARK-62071 (the classic modules of the objects of a 
session made PySpark's), five classic names still name the drop-in's native 
Connect classes, and the classic Window is a native class too:

- pyspark.sql.Column (pyspark.sql.column.Column) is 
pyspark.sql.connect.column.Column;
- pyspark.sql.DataFrameNaFunctions and DataFrameStatFunctions are the Connect 
classes of pyspark.sql.connect.dataframe;
- pyspark.sql.WindowSpec is pyspark.sql.connect.window.WindowSpec;
- pyspark.sql.table_arg.TableArg is pyspark.sql.connect.table_arg.TableArg;
- pyspark.sql.window.Window is a native class, not PySpark's classic class.

In PySpark 4.3.0, each classic class is a separate Python class, which its 
Connect class subclasses (python/pyspark/sql/connect/column.py: class 
Column(ParentColumn); connect/dataframe.py, connect/window.py, 
connect/table_arg.py likewise).

Repro, against a Spark 4.3.0 Connect server:

    import pyspark.sql.column, pyspark.sql.connect.column
    from pyspark.sql import Window, functions as F
    pyspark.sql.Column is pyspark.sql.connect.column.Column   # True; 
pyspark-client: False
    pyspark.sql.Column.__mro__        # (connect Column, object); 
pyspark-client: (Column, TableValuedFunctionArgument, object)
    pyspark.sql.Column.__module__     # 'pyspark.sql.connect.column'; 
pyspark-client: 'pyspark.sql.column'
    pyspark.sql.Column.alias(object(), "x")   # TypeError of a native 
descriptor; pyspark-client: NOT_IMPLEMENTED (dispatch_col_method)
    class MyColumn(pyspark.sql.Column): pass   # TypeError: type 'Column' is 
not an acceptable base type; pyspark-client: a subclass
    pyspark.sql.DataFrameNaFunctions(df)       # TypeError: cannot create ... 
instances; pyspark-client: an object of df
    pyspark.sql.dataframe.DataFrameNaFunctions.drop(df)   # TypeError; 
pyspark-client: df.drop() (dispatch_df_method)

The same holds for DataFrameStatFunctions, WindowSpec and TableArg; and 
pyspark.sql.tvf_argument.TableValuedFunctionArgument is an abc.ABC, of ABC in 
its MRO, where PySpark's is a plain class.

Expected, as in pyspark-client: the classic classes are PySpark's 
(python/pyspark/sql/column.py, table_arg.py, window.py and the classes of 
dataframe.py), of their modules, names, MROs, docstrings and dispatching 
methods, and the objects of a session are of the Connect classes, which are 
subclasses of them (for the drop-in's native classes, virtual ones), so that 
isinstance and issubclass hold, and a patch of a classic class reaches the 
objects as it does through PySpark's MRO.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to