Hyukjin Kwon created SPARK-62085:
------------------------------------
Summary: The types of pyspark.sql.types and other modules of
pyspark.sql and pyspark.errors have docstrings of their own, not PySpark's, in
the Python client
Key: SPARK-62085
URL: https://issues.apache.org/jira/browse/SPARK-62085
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by
their exact text (SHA-256). It covered the modules of pyspark.sql (but
pyspark.sql.functions, of SPARK-62074's follow-up) and of pyspark.errors that
both import, 121 of them:
- the module itself;
- each public class and function that the module defines;
- each public member (and `__init__`) of such a class, as Python finds it on
the class.
Of 4978 objects of PySpark, 2511 were the same, 1424 differed, and 1043 were
names that the drop-in does not have.
1. pyspark.sql.types (python/pyspark/sql/types.py): 308 differed. The types are
native, of the Rust extension, and their docstrings were their own:
- `StringType.__doc__`: PySpark's "String data type.\n\n Parameters\n
----------\n collation : str ..."; the drop-in's its own.
- `IntegerType.fromInternal.__doc__`: PySpark's that of
`DataType.fromInternal` ("\n Converts an internal SQL object into a
native Python object.\n "); the drop-in's its own.
- `DataType.json`, `jsonValue`, `simpleString` and `typeName`, and the
module: PySpark's have no docstring (None); the drop-in's had docstrings.
- UserDefinedType, a Python class: its own short docstrings.
2. The modules of pyspark.sql.connect, of the avro and protobuf functions, and
of pyspark.errors:
- 27 module docstrings were the drop-in's own, or present where PySpark's
have none;
- the SQLContext of pyspark.sql.connect.context (PySpark Connect's own class
of 4.3.0) differed in its docstring and those of its 44 members;
- `from_avro`, `to_avro`, `from_protobuf` and `to_protobuf` had short
docstrings of their own;
- a few classes and members differed: DataSourceRegistration,
GroupedData.mean (an alias of avg in PySpark), and the class docstrings of
DataFrameWriter, DataFrameWriterV2, StreamingQuery, StreamingQueryManager and
DataStreamWriter, which are None in PySpark.
Repro: `help(pyspark.sql.types.StructType)` and `help(spark.read.json)` of each
client, or `sha256(obj.__doc__)` of each object.
Some cannot be PySpark's with docstrings alone:
- The `__init__` of a native class is CPython's slot, whose docstring is
CPython's.
- PySpark's GeographyType and GeometryType define their own fromInternal and
needConversion, of no docstring, which the drop-in's inherit from DataType.
- The modules of the DataFrame, the Column, the window, the table argument, the
session and its client, and of the Connect functions, are of other work.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]