[ 
https://issues.apache.org/jira/browse/SPARK-62007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62007:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make patches of pyspark.sql.DataFrame reach native frames as PySpark's, take 
> PySpark's Arrow path for pandas data of struct subclasses and NUL names, and 
> fail names that UTF-8 cannot encode at the request
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62007
>                 URL: https://issues.apache.org/jira/browse/SPARK-62007
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> From a review of the documented differences of DataFrames against PySpark 
> 4.3.0, and the other session's review of the last 100 commits:
> - A patch of an attribute of pyspark.sql.DataFrame, such as DataFrame.count = 
> wrapper or mock.patch.object(DataFrame, "count"), reaches the drop-in's 
> native frames, where PySpark's Connect frames define every public name 
> themselves and see a patch of the base only through DataFrame.m(df). A 
> wrapper of the base and one of pyspark.sql.connect.dataframe.DataFrame that 
> call their saved originals make df.count() raise RecursionError, where 
> PySpark returns the count; a patch of __getattr__ reaches the frames too.
> - pandas createDataFrame with a StructType or StructField subclass, or a 
> column name that holds "\x00", takes the row path even with PyArrow, where 
> PySpark takes its Arrow path, so 2**53+1 under a DoubleType field converts 
> instead of raising, nanoseconds are truncated silently, and errors are other 
> classes.
> - createDataFrame of column names that UTF-8 cannot encode fails at the call, 
> where PySpark fails at the first request, through toDF.
> - The known differences of the surrogate names, the batch reader's option of 
> None, the row paths without PyArrow, the invalid UTF-8 of toArrow and 
> toPandas, the depth of plans that the server takes, and StorageLevel pickles 
> are stale or missing.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to