Hyukjin Kwon created SPARK-62101:
------------------------------------
Summary: help() of the Connect Column, WindowSpec, TableArg,
DataFrameNaFunctions and DataFrameStatFunctions does not show PySpark's
docstrings, and the Connect functions modules have docstrings of their own, in
the Python client
Key: SPARK-62101
URL: https://issues.apache.org/jira/browse/SPARK-62101
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by
their exact text (SHA-256), on master daacf3d1. It covered the modules of
pyspark.sql and pyspark.errors that both import, 124 of them:
- the module itself;
- each public class and function that the module defines;
- each public member (and `__init__`) of such a class, as Python finds it on
the class.
An object is compared by its `__doc__`. Where PySpark's object has no docstring
of its own and inherits one, it is compared by its `inspect.getdoc`, which is
what `help()` shows.
SPARK-62084 made pyspark.sql.Column, DataFrameNaFunctions,
DataFrameStatFunctions, Window, WindowSpec and TableArg PySpark's classic
classes, of which the native Connect classes are virtual subclasses. Their
modules (pyspark.sql.column, pyspark.sql.window, pyspark.sql.table_arg) are now
PySpark's. Those of Spark Connect still differ, in 65 docstrings:
| Module | Differ | What |
|---|---|---|
| pyspark.sql.connect.column | 39 | the Column class and 34 members of
inherited docstrings (alias, between, cast, over, when, ...), astype, name and
to_plan of none, and the module |
| pyspark.sql.connect.window | 6 | the WindowSpec class, orderBy, partitionBy,
rangeBetween, rowsBetween, of inherited docstrings, and the module |
| pyspark.sql.connect.table_arg | 6 | the TableArg class, orderBy, partitionBy,
withSinglePartition, of inherited docstrings, to_plan of none, and the module |
| pyspark.sql.connect.dataframe | 11 | DataFrameNaFunctions (the class, drop,
fill, replace) and DataFrameStatFunctions (the class, approxQuantile, corr,
cov, crosstab, freqItems, sampleBy), of inherited docstrings |
| pyspark.sql.connect.functions | 1 | the module |
| pyspark.sql.connect.functions.builtin | 1 | the module |
| pyspark.sql.connect.plan | 1 | Read.print, of an inherited docstring |
PySpark Connect's classes of these modules are subclasses of the classic ones,
and neither they nor these members have docstrings of their own. So their
`__doc__` is None, and `inspect.getdoc` and `help()` show the docstring of the
classic class, from the MRO. PySpark Connect's module docstrings there are
None, except that of pyspark.sql.connect.functions ("PySpark Functions with
Spark Connect").
The drop-in's classes are native and are virtual subclasses of the classic
ones, which are not in their MRO. Their docstrings were their own doc comments,
or none, and its modules had docstrings of their own. PySpark Connect's
`Read.print` inherits the docstring of `LogicalPlan.print`, which the drop-in's
`LogicalPlan` does not have.
Repro, without a server:
- `inspect.getdoc(pyspark.sql.connect.column.Column.alias)` is the classic
`Column.alias`'s docstring ("Returns this column aliased with a new name or
names ...") in pyspark-client, and the drop-in's own text in the drop-in.
- `inspect.getdoc(pyspark.sql.connect.dataframe.DataFrameNaFunctions.fill)` is
the classic `DataFrame.fillna`'s docstring in pyspark-client, and the drop-in's
own text in the drop-in.
- `inspect.getdoc(pyspark.sql.connect.column.Column.between)` is the classic
`Column.between`'s docstring in pyspark-client, and None in the drop-in.
- `pyspark.sql.connect.functions.__doc__` is "PySpark Functions with Spark
Connect" in pyspark-client, and the drop-in's own text in the drop-in.
What cannot be PySpark Connect's with docstrings alone:
- A native class cannot inherit a docstring from the classic class, as a
virtual subclass. To show PySpark's text in `help()`, its members carry the
classic docstrings, so their `__doc__` is that text, where PySpark Connect's is
None.
- The `__init__` of a native class is CPython's slot, whose docstring is
CPython's. `help()` shows the same text in pyspark-client, where PySpark
Connect's `__init__` has no docstring.
- These are of other work: the DataFrame class of both DataFrame modules, the
module docstrings of those modules, the 9 members of the classic
DataFrameNaFunctions and DataFrameStatFunctions that copy the DataFrame's
(`drop.__doc__ = DataFrame.dropna.__doc__`), and the modules of the session and
its client (pyspark.sql.connect.session, pyspark.sql.connect.client).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]