Hyukjin Kwon created SPARK-62080:
------------------------------------
Summary: The catalog, conf, readers and writers, grouped data,
merge writer, Observation and TVF classes lack the docstrings that PySpark
Connect copies of PySpark's classic classes, in the Python client
Key: SPARK-62080
URL: https://issues.apache.org/jira/browse/SPARK-62080
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
PySpark Connect's modules give their classes and members the docstrings of
PySpark's classic classes, as SPARK-62078 did for its UDF and UDTF
registrations and streaming classes, in 145 more assignments of these modules:
| Module (python/pyspark/sql/connect/) | Class | Docstrings |
|---|---|---|
| catalog.py | Catalog | 37 members and the class |
| conf.py | RuntimeConf | 6 members and the class, of the classic RuntimeConfig
|
| datasource.py | DataSourceRegistration | register |
| group.py | GroupedData | 14 members and the class |
| group.py | PandasCogroupedOps | 2 members and the class, of
pyspark.sql.pandas.group_ops |
| merge.py | MergeIntoWriter | 5 members, the class, and its WhenMatched,
WhenNotMatched and WhenNotMatchedBySource (8 members, 3 classes) |
| observation.py | Observation | 3 members and the class |
| readwriter.py | DataFrameReader | 14 members and the class |
| readwriter.py | DataFrameWriter | 18 members |
| readwriter.py | DataFrameWriterV2 | 12 members |
| tvf.py | TableValuedFunction | 14 members and the class |
The drop-in's classes of these modules are native, and all 145 of their
docstrings differ from pyspark-client's, of their own text or none. Repro:
- `type(spark.catalog).listTables.__doc__`: PySpark's 1343 characters; the
drop-in's 84.
- `type(spark.conf).get.__doc__`: 1846; the drop-in's 306.
- `type(spark.range(1).write).option.__doc__`: 1193; the drop-in's None.
- `pyspark.sql.connect.observation.Observation.__doc__`: 1611; the drop-in's
466.
Of them, the `__init__` of RuntimeConf and of Observation (conf.py:34,
observation.py) cannot be PySpark's in the drop-in: the `__init__` of a native
class is the slot of CPython, whose wrapper has CPython's docstring
("Initialize self. See help(type(self)) for accurate signature.").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]