[ 
https://issues.apache.org/jira/browse/SPARK-62018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62018:
------------------------------------

    Assignee: Hyukjin Kwon

> Keep in the type stubs the docstrings that PySpark's objects have at run 
> time, so that editors show PySpark's documentation as with pyspark-client
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62018
>                 URL: https://issues.apache.org/jira/browse/SPARK-62018
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> The drop-in for pyspark-client 4.3.0 ships type stubs (.pyi) of the modules 
> whose classes are native or whose names are made at import time 
> (scripts/gen_type_stubs.py, SPARK-61854 and SPARK-62011), which type checkers 
> and editors read instead of the modules. The stubs dropped every docstring, 
> so an editor showed the signature of a function or method of these modules 
> without its documentation.
> Repro: a hover of pyright 1.1.414's language server over these names, of 
> `from pyspark.sql import functions as F; from pyspark.sql.dataframe import 
> DataFrame; from pyspark.sql.column import Column`:
> - `F.col`: pyspark-client shows 416 characters, the signature and PySpark's 
> docstring ("Returns a :class:`~pyspark.sql.Column` based on the given column 
> name. ..."); the drop-in showed 38, the signature only.
> - `DataFrame.select`: 1097 characters with pyspark-client, the signatures 
> only with the drop-in.
> - `Column.cast`: 828 characters with pyspark-client, the signature only with 
> the drop-in.
> - The classes of pyspark.ml.connect: no docstring.
> Expected: the docstring of each object as PySpark has it at run time 
> (inspect.getdoc of PySpark's object), which is not always the docstring of 
> its source:
> - pyspark.since appends `.. versionadded::` (python/pyspark/__init__.py, 
> since);
> - the modules of PySpark Connect copy the docstrings of the classic ones 
> (`col.__doc__ = pysparkfuncs.col.__doc__`, 
> python/pyspark/sql/connect/functions/builtin.py; `__doc__ = 
> PySparkTableValuedFunction.__doc__` in a class body, connect/tvf.py);
> - a method of no docstring has that of its base (inspect.getdoc).
> Also, scripts/gen_type_stubs.py takes about 41 s of CPU to generate the 
> stubs, mostly in ast.get_source_segment, which splits the whole source of 
> PySpark's module again for each default value of a parameter that it renders.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to