[
https://issues.apache.org/jira/browse/SPARK-62085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62085.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 890
[https://github.com/apache/spark-connect-rust/pull/890]
> The types of pyspark.sql.types and other modules of pyspark.sql and
> pyspark.errors have docstrings of their own, not PySpark's, in the Python
> client
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62085
> URL: https://issues.apache.org/jira/browse/SPARK-62085
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> A sweep compared the docstrings of pyspark-client 4.3.0 and of the drop-in by
> their exact text (SHA-256). It covered the modules of pyspark.sql (but
> pyspark.sql.functions, of SPARK-62074's follow-up) and of pyspark.errors that
> both import, 121 of them:
> - the module itself;
> - each public class and function that the module defines;
> - each public member (and `__init__`) of such a class, as Python finds it on
> the class.
> Of 4978 objects of PySpark, 2511 were the same, 1424 differed, and 1043 were
> names that the drop-in does not have.
> 1. pyspark.sql.types (python/pyspark/sql/types.py): 308 differed. The types
> are native, of the Rust extension, and their docstrings were their own:
> - `StringType.__doc__`: PySpark's "String data type.\n\n Parameters\n
> ----------\n collation : str ..."; the drop-in's its own.
> - `IntegerType.fromInternal.__doc__`: PySpark's that of
> `DataType.fromInternal` ("\n Converts an internal SQL object into a
> native Python object.\n "); the drop-in's its own.
> - `DataType.json`, `jsonValue`, `simpleString` and `typeName`, and the
> module: PySpark's have no docstring (None); the drop-in's had docstrings.
> - UserDefinedType, a Python class: its own short docstrings.
> 2. The modules of pyspark.sql.connect, of the avro and protobuf functions,
> and of pyspark.errors:
> - 27 module docstrings were the drop-in's own, or present where PySpark's
> have none;
> - the SQLContext of pyspark.sql.connect.context (PySpark Connect's own
> class of 4.3.0) differed in its docstring and those of its 44 members;
> - `from_avro`, `to_avro`, `from_protobuf` and `to_protobuf` had short
> docstrings of their own;
> - a few classes and members differed: DataSourceRegistration,
> GroupedData.mean (an alias of avg in PySpark), and the class docstrings of
> DataFrameWriter, DataFrameWriterV2, StreamingQuery, StreamingQueryManager and
> DataStreamWriter, which are None in PySpark.
> Repro: `help(pyspark.sql.types.StructType)` and `help(spark.read.json)` of
> each client, or `sha256(obj.__doc__)` of each object.
> Some cannot be PySpark's with docstrings alone:
> - The `__init__` of a native class is CPython's slot, whose docstring is
> CPython's.
> - PySpark's GeographyType and GeometryType define their own fromInternal and
> needConversion, of no docstring, which the drop-in's inherit from DataType.
> - The modules of the DataFrame, the Column, the window, the table argument,
> the session and its client, and of the Connect functions, are of other work.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]