[
https://issues.apache.org/jira/browse/SPARK-62101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62101:
-----------------------------------
Labels: pull-request-available (was: )
> help() of the Connect Column, WindowSpec, TableArg, DataFrameNaFunctions and
> DataFrameStatFunctions does not show PySpark's docstrings, and the Connect
> functions modules have docstrings of their own, in the Python client
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62101
> URL: https://issues.apache.org/jira/browse/SPARK-62101
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by
> their exact text (SHA-256), on master daacf3d1. It covered the modules of
> pyspark.sql and pyspark.errors that both import, 124 of them:
> - the module itself;
> - each public class and function that the module defines;
> - each public member (and `__init__`) of such a class, as Python finds it on
> the class.
> An object is compared by its `__doc__`. Where PySpark's object has no
> docstring of its own and inherits one, it is compared by its
> `inspect.getdoc`, which is what `help()` shows.
> SPARK-62084 made pyspark.sql.Column, DataFrameNaFunctions,
> DataFrameStatFunctions, Window, WindowSpec and TableArg PySpark's classic
> classes, of which the native Connect classes are virtual subclasses. Their
> modules (pyspark.sql.column, pyspark.sql.window, pyspark.sql.table_arg) are
> now PySpark's. Those of Spark Connect still differ, in 65 docstrings:
> | Module | Differ | What |
> |---|---|---|
> | pyspark.sql.connect.column | 39 | the Column class and 34 members of
> inherited docstrings (alias, between, cast, over, when, ...), astype, name
> and to_plan of none, and the module |
> | pyspark.sql.connect.window | 6 | the WindowSpec class, orderBy,
> partitionBy, rangeBetween, rowsBetween, of inherited docstrings, and the
> module |
> | pyspark.sql.connect.table_arg | 6 | the TableArg class, orderBy,
> partitionBy, withSinglePartition, of inherited docstrings, to_plan of none,
> and the module |
> | pyspark.sql.connect.dataframe | 11 | DataFrameNaFunctions (the class, drop,
> fill, replace) and DataFrameStatFunctions (the class, approxQuantile, corr,
> cov, crosstab, freqItems, sampleBy), of inherited docstrings |
> | pyspark.sql.connect.functions | 1 | the module |
> | pyspark.sql.connect.functions.builtin | 1 | the module |
> | pyspark.sql.connect.plan | 1 | Read.print, of an inherited docstring |
> PySpark Connect's classes of these modules are subclasses of the classic
> ones, and neither they nor these members have docstrings of their own. So
> their `__doc__` is None, and `inspect.getdoc` and `help()` show the docstring
> of the classic class, from the MRO. PySpark Connect's module docstrings there
> are None, except that of pyspark.sql.connect.functions ("PySpark Functions
> with Spark Connect").
> The drop-in's classes are native and are virtual subclasses of the classic
> ones, which are not in their MRO. Their docstrings were their own doc
> comments, or none, and its modules had docstrings of their own. PySpark
> Connect's `Read.print` inherits the docstring of `LogicalPlan.print`, which
> the drop-in's `LogicalPlan` does not have.
> Repro, without a server:
> - `inspect.getdoc(pyspark.sql.connect.column.Column.alias)` is the classic
> `Column.alias`'s docstring ("Returns this column aliased with a new name or
> names ...") in pyspark-client, and the drop-in's own text in the drop-in.
> - `inspect.getdoc(pyspark.sql.connect.dataframe.DataFrameNaFunctions.fill)`
> is the classic `DataFrame.fillna`'s docstring in pyspark-client, and the
> drop-in's own text in the drop-in.
> - `inspect.getdoc(pyspark.sql.connect.column.Column.between)` is the classic
> `Column.between`'s docstring in pyspark-client, and None in the drop-in.
> - `pyspark.sql.connect.functions.__doc__` is "PySpark Functions with Spark
> Connect" in pyspark-client, and the drop-in's own text in the drop-in.
> What cannot be PySpark Connect's with docstrings alone:
> - A native class cannot inherit a docstring from the classic class, as a
> virtual subclass. To show PySpark's text in `help()`, its members carry the
> classic docstrings, so their `__doc__` is that text, where PySpark Connect's
> is None.
> - The `__init__` of a native class is CPython's slot, whose docstring is
> CPython's. `help()` shows the same text in pyspark-client, where PySpark
> Connect's `__init__` has no docstring.
> - These are of other work: the DataFrame class of both DataFrame modules, the
> module docstrings of those modules, the 9 members of the classic
> DataFrameNaFunctions and DataFrameStatFunctions that copy the DataFrame's
> (`drop.__doc__ = DataFrame.dropna.__doc__`), and the modules of the session
> and its client (pyspark.sql.connect.session, pyspark.sql.connect.client).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]