[ 
https://issues.apache.org/jira/browse/SPARK-62085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62085:
------------------------------------

    Assignee: Hyukjin Kwon

> The types of pyspark.sql.types and other modules of pyspark.sql and 
> pyspark.errors have docstrings of their own, not PySpark's, in the Python 
> client
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62085
>                 URL: https://issues.apache.org/jira/browse/SPARK-62085
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by 
> their exact text (SHA-256). It covered the modules of pyspark.sql (but 
> pyspark.sql.functions, of SPARK-62074's follow-up) and of pyspark.errors that 
> both import, 121 of them:
> - the module itself;
> - each public class and function that the module defines;
> - each public member (and `__init__`) of such a class, as Python finds it on 
> the class.
> Of 4978 objects of PySpark, 2511 were the same, 1424 differed, and 1043 were 
> names that the drop-in does not have.
> 1. pyspark.sql.types (python/pyspark/sql/types.py): 308 differed. The types 
> are native, of the Rust extension, and their docstrings were their own:
>    - `StringType.__doc__`: PySpark's "String data type.\n\n    Parameters\n   
>  ----------\n    collation : str ..."; the drop-in's its own.
>    - `IntegerType.fromInternal.__doc__`: PySpark's that of 
> `DataType.fromInternal` ("\n        Converts an internal SQL object into a 
> native Python object.\n        "); the drop-in's its own.
>    - `DataType.json`, `jsonValue`, `simpleString` and `typeName`, and the 
> module: PySpark's have no docstring (None); the drop-in's had docstrings.
>    - UserDefinedType, a Python class: its own short docstrings.
> 2. The modules of pyspark.sql.connect, of the avro and protobuf functions, 
> and of pyspark.errors:
>    - 27 module docstrings were the drop-in's own, or present where PySpark's 
> have none;
>    - the SQLContext of pyspark.sql.connect.context (PySpark Connect's own 
> class of 4.3.0) differed in its docstring and those of its 44 members;
>    - `from_avro`, `to_avro`, `from_protobuf` and `to_protobuf` had short 
> docstrings of their own;
>    - a few classes and members differed: DataSourceRegistration, 
> GroupedData.mean (an alias of avg in PySpark), and the class docstrings of 
> DataFrameWriter, DataFrameWriterV2, StreamingQuery, StreamingQueryManager and 
> DataStreamWriter, which are None in PySpark.
> Repro: `help(pyspark.sql.types.StructType)` and `help(spark.read.json)` of 
> each client, or `sha256(obj.__doc__)` of each object.
> Some cannot be PySpark's with docstrings alone:
> - The `__init__` of a native class is CPython's slot, whose docstring is 
> CPython's.
> - PySpark's GeographyType and GeometryType define their own fromInternal and 
> needConversion, of no docstring, which the drop-in's inherit from DataType.
> - The modules of the DataFrame, the Column, the window, the table argument, 
> the session and its client, and of the Connect functions, are of other work.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to