Hyukjin Kwon created SPARK-62007:
------------------------------------
Summary: Make patches of pyspark.sql.DataFrame reach native frames
as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses
and NUL names, and fail names that UTF-8 cannot encode at the request
Key: SPARK-62007
URL: https://issues.apache.org/jira/browse/SPARK-62007
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
>From a review of the documented differences of DataFrames against PySpark
>4.3.0, and the other session's review of the last 100 commits:
- A patch of an attribute of pyspark.sql.DataFrame, such as DataFrame.count =
wrapper or mock.patch.object(DataFrame, "count"), reaches the drop-in's native
frames, where PySpark's Connect frames define every public name themselves and
see a patch of the base only through DataFrame.m(df). A wrapper of the base and
one of pyspark.sql.connect.dataframe.DataFrame that call their saved originals
make df.count() raise RecursionError, where PySpark returns the count; a patch
of __getattr__ reaches the frames too.
- pandas createDataFrame with a StructType or StructField subclass, or a column
name that holds "\x00", takes the row path even with PyArrow, where PySpark
takes its Arrow path, so 2**53+1 under a DoubleType field converts instead of
raising, nanoseconds are truncated silently, and errors are other classes.
- createDataFrame of column names that UTF-8 cannot encode fails at the call,
where PySpark fails at the first request, through toDF.
- The known differences of the surrogate names, the batch reader's option of
None, the row paths without PyArrow, the invalid UTF-8 of toArrow and toPandas,
the depth of plans that the server takes, and StorageLevel pickles are stale or
missing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]