[
https://issues.apache.org/jira/browse/SPARK-62084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62084.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 891
[https://github.com/apache/spark-connect-rust/pull/891]
> Make pyspark.sql.Column, DataFrameNaFunctions, DataFrameStatFunctions,
> Window, WindowSpec and TableArg PySpark's classic classes, of which the
> native Connect classes are virtual subclasses, as in pyspark-client
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62084
> URL: https://issues.apache.org/jira/browse/SPARK-62084
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> The drop-in for pyspark-client 4.3.0 gives the objects of a session PySpark
> Connect's classes. After SPARK-62071 (the classic modules of the objects of a
> session made PySpark's), five classic names still name the drop-in's native
> Connect classes, and the classic Window is a native class too:
> - pyspark.sql.Column (pyspark.sql.column.Column) is
> pyspark.sql.connect.column.Column;
> - pyspark.sql.DataFrameNaFunctions and DataFrameStatFunctions are the Connect
> classes of pyspark.sql.connect.dataframe;
> - pyspark.sql.WindowSpec is pyspark.sql.connect.window.WindowSpec;
> - pyspark.sql.table_arg.TableArg is pyspark.sql.connect.table_arg.TableArg;
> - pyspark.sql.window.Window is a native class, not PySpark's classic class.
> In PySpark 4.3.0, each classic class is a separate Python class, which its
> Connect class subclasses (python/pyspark/sql/connect/column.py: class
> Column(ParentColumn); connect/dataframe.py, connect/window.py,
> connect/table_arg.py likewise).
> Repro, against a Spark 4.3.0 Connect server:
> import pyspark.sql.column, pyspark.sql.connect.column
> from pyspark.sql import Window, functions as F
> pyspark.sql.Column is pyspark.sql.connect.column.Column # True;
> pyspark-client: False
> pyspark.sql.Column.__mro__ # (connect Column, object);
> pyspark-client: (Column, TableValuedFunctionArgument, object)
> pyspark.sql.Column.__module__ # 'pyspark.sql.connect.column';
> pyspark-client: 'pyspark.sql.column'
> pyspark.sql.Column.alias(object(), "x") # TypeError of a native
> descriptor; pyspark-client: NOT_IMPLEMENTED (dispatch_col_method)
> class MyColumn(pyspark.sql.Column): pass # TypeError: type 'Column' is
> not an acceptable base type; pyspark-client: a subclass
> pyspark.sql.DataFrameNaFunctions(df) # TypeError: cannot create ...
> instances; pyspark-client: an object of df
> pyspark.sql.dataframe.DataFrameNaFunctions.drop(df) # TypeError;
> pyspark-client: df.drop() (dispatch_df_method)
> The same holds for DataFrameStatFunctions, WindowSpec and TableArg; and
> pyspark.sql.tvf_argument.TableValuedFunctionArgument is an abc.ABC, of ABC in
> its MRO, where PySpark's is a plain class.
> Expected, as in pyspark-client: the classic classes are PySpark's
> (python/pyspark/sql/column.py, table_arg.py, window.py and the classes of
> dataframe.py), of their modules, names, MROs, docstrings and dispatching
> methods, and the objects of a session are of the Connect classes, which are
> subclasses of them (for the drop-in's native classes, virtual ones), so that
> isinstance and issubclass hold, and a patch of a classic class reaches the
> objects as it does through PySpark's MRO.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]