Hyukjin Kwon created SPARK-62101:
------------------------------------

             Summary: help() of the Connect Column, WindowSpec, TableArg, 
DataFrameNaFunctions and DataFrameStatFunctions does not show PySpark's 
docstrings, and the Connect functions modules have docstrings of their own, in 
the Python client
                 Key: SPARK-62101
                 URL: https://issues.apache.org/jira/browse/SPARK-62101
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by 
their exact text (SHA-256), on master daacf3d1. It covered the modules of 
pyspark.sql and pyspark.errors that both import, 124 of them:
- the module itself;
- each public class and function that the module defines;
- each public member (and `__init__`) of such a class, as Python finds it on 
the class.

An object is compared by its `__doc__`. Where PySpark's object has no docstring 
of its own and inherits one, it is compared by its `inspect.getdoc`, which is 
what `help()` shows.

SPARK-62084 made pyspark.sql.Column, DataFrameNaFunctions, 
DataFrameStatFunctions, Window, WindowSpec and TableArg PySpark's classic 
classes, of which the native Connect classes are virtual subclasses. Their 
modules (pyspark.sql.column, pyspark.sql.window, pyspark.sql.table_arg) are now 
PySpark's. Those of Spark Connect still differ, in 65 docstrings:

| Module | Differ | What |
|---|---|---|
| pyspark.sql.connect.column | 39 | the Column class and 34 members of 
inherited docstrings (alias, between, cast, over, when, ...), astype, name and 
to_plan of none, and the module |
| pyspark.sql.connect.window | 6 | the WindowSpec class, orderBy, partitionBy, 
rangeBetween, rowsBetween, of inherited docstrings, and the module |
| pyspark.sql.connect.table_arg | 6 | the TableArg class, orderBy, partitionBy, 
withSinglePartition, of inherited docstrings, to_plan of none, and the module |
| pyspark.sql.connect.dataframe | 11 | DataFrameNaFunctions (the class, drop, 
fill, replace) and DataFrameStatFunctions (the class, approxQuantile, corr, 
cov, crosstab, freqItems, sampleBy), of inherited docstrings |
| pyspark.sql.connect.functions | 1 | the module |
| pyspark.sql.connect.functions.builtin | 1 | the module |
| pyspark.sql.connect.plan | 1 | Read.print, of an inherited docstring |

PySpark Connect's classes of these modules are subclasses of the classic ones, 
and neither they nor these members have docstrings of their own. So their 
`__doc__` is None, and `inspect.getdoc` and `help()` show the docstring of the 
classic class, from the MRO. PySpark Connect's module docstrings there are 
None, except that of pyspark.sql.connect.functions ("PySpark Functions with 
Spark Connect").

The drop-in's classes are native and are virtual subclasses of the classic 
ones, which are not in their MRO. Their docstrings were their own doc comments, 
or none, and its modules had docstrings of their own. PySpark Connect's 
`Read.print` inherits the docstring of `LogicalPlan.print`, which the drop-in's 
`LogicalPlan` does not have.

Repro, without a server:
- `inspect.getdoc(pyspark.sql.connect.column.Column.alias)` is the classic 
`Column.alias`'s docstring ("Returns this column aliased with a new name or 
names ...") in pyspark-client, and the drop-in's own text in the drop-in.
- `inspect.getdoc(pyspark.sql.connect.dataframe.DataFrameNaFunctions.fill)` is 
the classic `DataFrame.fillna`'s docstring in pyspark-client, and the drop-in's 
own text in the drop-in.
- `inspect.getdoc(pyspark.sql.connect.column.Column.between)` is the classic 
`Column.between`'s docstring in pyspark-client, and None in the drop-in.
- `pyspark.sql.connect.functions.__doc__` is "PySpark Functions with Spark 
Connect" in pyspark-client, and the drop-in's own text in the drop-in.

What cannot be PySpark Connect's with docstrings alone:
- A native class cannot inherit a docstring from the classic class, as a 
virtual subclass. To show PySpark's text in `help()`, its members carry the 
classic docstrings, so their `__doc__` is that text, where PySpark Connect's is 
None.
- The `__init__` of a native class is CPython's slot, whose docstring is 
CPython's. `help()` shows the same text in pyspark-client, where PySpark 
Connect's `__init__` has no docstring.
- These are of other work: the DataFrame class of both DataFrame modules, the 
module docstrings of those modules, the 9 members of the classic 
DataFrameNaFunctions and DataFrameStatFunctions that copy the DataFrame's 
(`drop.__doc__ = DataFrame.dropna.__doc__`), and the modules of the session and 
its client (pyspark.sql.connect.session, pyspark.sql.connect.client).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to