Hyukjin Kwon created SPARK-62018:
------------------------------------

             Summary: Keep in the type stubs the docstrings that PySpark's 
objects have at run time, so that editors show PySpark's documentation as with 
pyspark-client
                 Key: SPARK-62018
                 URL: https://issues.apache.org/jira/browse/SPARK-62018
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The drop-in for pyspark-client 4.3.0 ships type stubs (.pyi) of the modules 
whose classes are native or whose names are made at import time 
(scripts/gen_type_stubs.py, SPARK-61854 and SPARK-62011), which type checkers 
and editors read instead of the modules. The stubs dropped every docstring, so 
an editor showed the signature of a function or method of these modules without 
its documentation.

Repro: a hover of pyright 1.1.414's language server over these names, of `from 
pyspark.sql import functions as F; from pyspark.sql.dataframe import DataFrame; 
from pyspark.sql.column import Column`:
- `F.col`: pyspark-client shows 416 characters, the signature and PySpark's 
docstring ("Returns a :class:`~pyspark.sql.Column` based on the given column 
name. ..."); the drop-in showed 38, the signature only.
- `DataFrame.select`: 1097 characters with pyspark-client, the signatures only 
with the drop-in.
- `Column.cast`: 828 characters with pyspark-client, the signature only with 
the drop-in.
- The classes of pyspark.ml.connect: no docstring.

Expected: the docstring of each object as PySpark has it at run time 
(inspect.getdoc of PySpark's object), which is not always the docstring of its 
source:
- pyspark.since appends `.. versionadded::` (python/pyspark/__init__.py, since);
- the modules of PySpark Connect copy the docstrings of the classic ones 
(`col.__doc__ = pysparkfuncs.col.__doc__`, 
python/pyspark/sql/connect/functions/builtin.py; `__doc__ = 
PySparkTableValuedFunction.__doc__` in a class body, connect/tvf.py);
- a method of no docstring has that of its base (inspect.getdoc).

Also, scripts/gen_type_stubs.py takes about 41 s of CPU to generate the stubs, 
mostly in ast.get_source_segment, which splits the whole source of PySpark's 
module again for each default value of a parameter that it renders.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to