[ 
https://issues.apache.org/jira/browse/SPARK-62080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62080.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 886
[https://github.com/apache/spark-connect-rust/pull/886]

> The catalog, conf, readers and writers, grouped data, merge writer, 
> Observation and TVF classes lack the docstrings that PySpark Connect copies 
> of PySpark's classic classes, in the Python client
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62080
>                 URL: https://issues.apache.org/jira/browse/SPARK-62080
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> PySpark Connect's modules give their classes and members the docstrings of 
> PySpark's classic classes, as SPARK-62078 did for its UDF and UDTF 
> registrations and streaming classes, in 145 more assignments of these modules:
> | Module (python/pyspark/sql/connect/) | Class | Docstrings |
> |---|---|---|
> | catalog.py | Catalog | 37 members and the class |
> | conf.py | RuntimeConf | 6 members and the class, of the classic 
> RuntimeConfig |
> | datasource.py | DataSourceRegistration | register |
> | group.py | GroupedData | 14 members and the class |
> | group.py | PandasCogroupedOps | 2 members and the class, of 
> pyspark.sql.pandas.group_ops |
> | merge.py | MergeIntoWriter | 5 members, the class, and its WhenMatched, 
> WhenNotMatched and WhenNotMatchedBySource (8 members, 3 classes) |
> | observation.py | Observation | 3 members and the class |
> | readwriter.py | DataFrameReader | 14 members and the class |
> | readwriter.py | DataFrameWriter | 18 members |
> | readwriter.py | DataFrameWriterV2 | 12 members |
> | tvf.py | TableValuedFunction | 14 members and the class |
> The drop-in's classes of these modules are native, and all 145 of their 
> docstrings differ from pyspark-client's, of their own text or none. Repro:
> - `type(spark.catalog).listTables.__doc__`: PySpark's 1343 characters; the 
> drop-in's 84.
> - `type(spark.conf).get.__doc__`: 1846; the drop-in's 306.
> - `type(spark.range(1).write).option.__doc__`: 1193; the drop-in's None.
> - `pyspark.sql.connect.observation.Observation.__doc__`: 1611; the drop-in's 
> 466.
> Of them, the `__init__` of RuntimeConf and of Observation (conf.py:34, 
> observation.py) cannot be PySpark's in the drop-in: the `__init__` of a 
> native class is the slot of CPython, whose wrapper has CPython's docstring 
> ("Initialize self.  See help(type(self)) for accurate signature.").



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to