[ 
https://issues.apache.org/jira/browse/SPARK-62084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62084:
------------------------------------

    Assignee: Hyukjin Kwon

> Make pyspark.sql.Column, DataFrameNaFunctions, DataFrameStatFunctions, 
> Window, WindowSpec and TableArg PySpark's classic classes, of which the 
> native Connect classes are virtual subclasses, as in pyspark-client
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62084
>                 URL: https://issues.apache.org/jira/browse/SPARK-62084
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> The drop-in for pyspark-client 4.3.0 gives the objects of a session PySpark 
> Connect's classes. After SPARK-62071 (the classic modules of the objects of a 
> session made PySpark's), five classic names still name the drop-in's native 
> Connect classes, and the classic Window is a native class too:
> - pyspark.sql.Column (pyspark.sql.column.Column) is 
> pyspark.sql.connect.column.Column;
> - pyspark.sql.DataFrameNaFunctions and DataFrameStatFunctions are the Connect 
> classes of pyspark.sql.connect.dataframe;
> - pyspark.sql.WindowSpec is pyspark.sql.connect.window.WindowSpec;
> - pyspark.sql.table_arg.TableArg is pyspark.sql.connect.table_arg.TableArg;
> - pyspark.sql.window.Window is a native class, not PySpark's classic class.
> In PySpark 4.3.0, each classic class is a separate Python class, which its 
> Connect class subclasses (python/pyspark/sql/connect/column.py: class 
> Column(ParentColumn); connect/dataframe.py, connect/window.py, 
> connect/table_arg.py likewise).
> Repro, against a Spark 4.3.0 Connect server:
>     import pyspark.sql.column, pyspark.sql.connect.column
>     from pyspark.sql import Window, functions as F
>     pyspark.sql.Column is pyspark.sql.connect.column.Column   # True; 
> pyspark-client: False
>     pyspark.sql.Column.__mro__        # (connect Column, object); 
> pyspark-client: (Column, TableValuedFunctionArgument, object)
>     pyspark.sql.Column.__module__     # 'pyspark.sql.connect.column'; 
> pyspark-client: 'pyspark.sql.column'
>     pyspark.sql.Column.alias(object(), "x")   # TypeError of a native 
> descriptor; pyspark-client: NOT_IMPLEMENTED (dispatch_col_method)
>     class MyColumn(pyspark.sql.Column): pass   # TypeError: type 'Column' is 
> not an acceptable base type; pyspark-client: a subclass
>     pyspark.sql.DataFrameNaFunctions(df)       # TypeError: cannot create ... 
> instances; pyspark-client: an object of df
>     pyspark.sql.dataframe.DataFrameNaFunctions.drop(df)   # TypeError; 
> pyspark-client: df.drop() (dispatch_df_method)
> The same holds for DataFrameStatFunctions, WindowSpec and TableArg; and 
> pyspark.sql.tvf_argument.TableValuedFunctionArgument is an abc.ABC, of ABC in 
> its MRO, where PySpark's is a plain class.
> Expected, as in pyspark-client: the classic classes are PySpark's 
> (python/pyspark/sql/column.py, table_arg.py, window.py and the classes of 
> dataframe.py), of their modules, names, MROs, docstrings and dispatching 
> methods, and the objects of a session are of the Connect classes, which are 
> subclasses of them (for the drop-in's native classes, virtual ones), so that 
> isinstance and issubclass hold, and a patch of a classic class reaches the 
> objects as it does through PySpark's MRO.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to