Hyukjin Kwon created SPARK-62080:
------------------------------------

             Summary: The catalog, conf, readers and writers, grouped data, 
merge writer, Observation and TVF classes lack the docstrings that PySpark 
Connect copies of PySpark's classic classes, in the Python client
                 Key: SPARK-62080
                 URL: https://issues.apache.org/jira/browse/SPARK-62080
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


PySpark Connect's modules give their classes and members the docstrings of 
PySpark's classic classes, as SPARK-62078 did for its UDF and UDTF 
registrations and streaming classes, in 145 more assignments of these modules:

| Module (python/pyspark/sql/connect/) | Class | Docstrings |
|---|---|---|
| catalog.py | Catalog | 37 members and the class |
| conf.py | RuntimeConf | 6 members and the class, of the classic RuntimeConfig 
|
| datasource.py | DataSourceRegistration | register |
| group.py | GroupedData | 14 members and the class |
| group.py | PandasCogroupedOps | 2 members and the class, of 
pyspark.sql.pandas.group_ops |
| merge.py | MergeIntoWriter | 5 members, the class, and its WhenMatched, 
WhenNotMatched and WhenNotMatchedBySource (8 members, 3 classes) |
| observation.py | Observation | 3 members and the class |
| readwriter.py | DataFrameReader | 14 members and the class |
| readwriter.py | DataFrameWriter | 18 members |
| readwriter.py | DataFrameWriterV2 | 12 members |
| tvf.py | TableValuedFunction | 14 members and the class |

The drop-in's classes of these modules are native, and all 145 of their 
docstrings differ from pyspark-client's, of their own text or none. Repro:
- `type(spark.catalog).listTables.__doc__`: PySpark's 1343 characters; the 
drop-in's 84.
- `type(spark.conf).get.__doc__`: 1846; the drop-in's 306.
- `type(spark.range(1).write).option.__doc__`: 1193; the drop-in's None.
- `pyspark.sql.connect.observation.Observation.__doc__`: 1611; the drop-in's 
466.

Of them, the `__init__` of RuntimeConf and of Observation (conf.py:34, 
observation.py) cannot be PySpark's in the drop-in: the `__init__` of a native 
class is the slot of CPython, whose wrapper has CPython's docstring 
("Initialize self.  See help(type(self)) for accurate signature.").




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to