Hyukjin Kwon created SPARK-62085:
------------------------------------

             Summary: The types of pyspark.sql.types and other modules of 
pyspark.sql and pyspark.errors have docstrings of their own, not PySpark's, in 
the Python client
                 Key: SPARK-62085
                 URL: https://issues.apache.org/jira/browse/SPARK-62085
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by 
their exact text (SHA-256). It covered the modules of pyspark.sql (but 
pyspark.sql.functions, of SPARK-62074's follow-up) and of pyspark.errors that 
both import, 121 of them:
- the module itself;
- each public class and function that the module defines;
- each public member (and `__init__`) of such a class, as Python finds it on 
the class.

Of 4978 objects of PySpark, 2511 were the same, 1424 differed, and 1043 were 
names that the drop-in does not have.

1. pyspark.sql.types (python/pyspark/sql/types.py): 308 differed. The types are 
native, of the Rust extension, and their docstrings were their own:
   - `StringType.__doc__`: PySpark's "String data type.\n\n    Parameters\n    
----------\n    collation : str ..."; the drop-in's its own.
   - `IntegerType.fromInternal.__doc__`: PySpark's that of 
`DataType.fromInternal` ("\n        Converts an internal SQL object into a 
native Python object.\n        "); the drop-in's its own.
   - `DataType.json`, `jsonValue`, `simpleString` and `typeName`, and the 
module: PySpark's have no docstring (None); the drop-in's had docstrings.
   - UserDefinedType, a Python class: its own short docstrings.
2. The modules of pyspark.sql.connect, of the avro and protobuf functions, and 
of pyspark.errors:
   - 27 module docstrings were the drop-in's own, or present where PySpark's 
have none;
   - the SQLContext of pyspark.sql.connect.context (PySpark Connect's own class 
of 4.3.0) differed in its docstring and those of its 44 members;
   - `from_avro`, `to_avro`, `from_protobuf` and `to_protobuf` had short 
docstrings of their own;
   - a few classes and members differed: DataSourceRegistration, 
GroupedData.mean (an alias of avg in PySpark), and the class docstrings of 
DataFrameWriter, DataFrameWriterV2, StreamingQuery, StreamingQueryManager and 
DataStreamWriter, which are None in PySpark.

Repro: `help(pyspark.sql.types.StructType)` and `help(spark.read.json)` of each 
client, or `sha256(obj.__doc__)` of each object.

Some cannot be PySpark's with docstrings alone:
- The `__init__` of a native class is CPython's slot, whose docstring is 
CPython's.
- PySpark's GeographyType and GeometryType define their own fromInternal and 
needConversion, of no docstring, which the drop-in's inherit from DataType.
- The modules of the DataFrame, the Column, the window, the table argument, the 
session and its client, and of the Connect functions, are of other work.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to