[
https://issues.apache.org/jira/browse/SPARK-62007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62007:
------------------------------------
Assignee: Hyukjin Kwon
> Make patches of pyspark.sql.DataFrame reach native frames as PySpark's, take
> PySpark's Arrow path for pandas data of struct subclasses and NUL names, and
> fail names that UTF-8 cannot encode at the request
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62007
> URL: https://issues.apache.org/jira/browse/SPARK-62007
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> From a review of the documented differences of DataFrames against PySpark
> 4.3.0, and the other session's review of the last 100 commits:
> - A patch of an attribute of pyspark.sql.DataFrame, such as DataFrame.count =
> wrapper or mock.patch.object(DataFrame, "count"), reaches the drop-in's
> native frames, where PySpark's Connect frames define every public name
> themselves and see a patch of the base only through DataFrame.m(df). A
> wrapper of the base and one of pyspark.sql.connect.dataframe.DataFrame that
> call their saved originals make df.count() raise RecursionError, where
> PySpark returns the count; a patch of __getattr__ reaches the frames too.
> - pandas createDataFrame with a StructType or StructField subclass, or a
> column name that holds "\x00", takes the row path even with PyArrow, where
> PySpark takes its Arrow path, so 2**53+1 under a DoubleType field converts
> instead of raising, nanoseconds are truncated silently, and errors are other
> classes.
> - createDataFrame of column names that UTF-8 cannot encode fails at the call,
> where PySpark fails at the first request, through toDF.
> - The known differences of the surrogate names, the batch reader's option of
> None, the row paths without PyArrow, the invalid UTF-8 of toArrow and
> toPandas, the depth of plans that the server takes, and StorageLevel pickles
> are stale or missing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]